The following have been reloaded with a version change: 1) GCCcore/.14.3.0 => GCCcore/14.3.0 Lmod is automatically replacing "GCC/14.3.0" with "nvidia-compilers/25.9-CUDA-13". Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python [ray] RAY_TMPDIR=/tmp/ray/ray_639476 [triton_cache] Triton cache: /tmp/triton_cache_feuer1_639476 [triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_639476 [proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 [proxy] Setting up SSH tunnel to jpbl-s01-01 [proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc [proxy] Tunnel port: 7003 [proxy] Node IP: 10.128.32.49 (workers will connect here) [proxy] ✓ SSH tunnel started successfully [proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_639476.conf [proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT [proxy] - External traffic (internet) → PROXY via tunnel [proxy] ✓ Daytona timeout settings configured [proxy] Testing proxy connectivity... [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_639476.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary) [proxy] ⚠ Tunnel not accessible at 10.128.32.49:7003 (workers may fail) [proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers) [container_runtime] Using cloud backend: daytona (no local container setup) === Universal RL Training Runner === Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/configs/ablation-pymethods2test-seqnorm_rl_config.json Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python version: Python 3.12.12 UV_USE_IO_URING: 0 Proxy: DISABLED (direct internet or not configured) ======================================== === RLJobRunner: ablation-pymethods2test-seqnorm === [wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/wandb [wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/wandb HF_TOKEN=****pDbg HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config) Environment configured: TENSOR_PARALLEL_SIZE=1 NUM_INFERENCE_ENGINES=56 POLICY_NUM_NODES=14 WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/wandb Starting Ray cluster with 14 nodes, 4 GPUs/node Cleaning up existing Ray instances... === Starting Ray Cluster === Nodes: 14 GPUs per node: 4 CPUs per node: 288 Head node: jpbo-042-01 (10.128.32.49) Ray port: 6379 ============================ Starting Ray head on jpbo-042-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-042-01.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.49 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.49 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray head on jpbo-042-01 Starting Ray worker on jpbo-042-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-02.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 1 on jpbo-042-02 Starting Ray worker on jpbo-042-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-03.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 2 on jpbo-042-03 Starting Ray worker on jpbo-042-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-04.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 3 on jpbo-042-04 Starting Ray worker on jpbo-042-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-05.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 4 on jpbo-042-05 Starting Ray worker on jpbo-042-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-06.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.54 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.54 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 5 on jpbo-042-06 Starting Ray worker on jpbo-042-07 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-07.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.55 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.55 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 6 on jpbo-042-07 Starting Ray worker on jpbo-042-08 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-08.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.56 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.56 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 7 on jpbo-042-08 Starting Ray worker on jpbo-042-09 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-09.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.57 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-09 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.57 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 8 on jpbo-042-09 Starting Ray worker on jpbo-042-10 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-10.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.58 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-10 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.58 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 9 on jpbo-042-10 Starting Ray worker on jpbo-042-11 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-11.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.59 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.59 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 10 on jpbo-042-11 Starting Ray worker on jpbo-042-12 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-12.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.60 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.60 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 11 on jpbo-042-12 Starting Ray worker on jpbo-042-13 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-13.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.61 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.61 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 12 on jpbo-042-13 Starting Ray worker on jpbo-042-14 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-14.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.62 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-14 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.62 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 13 on jpbo-042-14 Waiting for cluster (56 GPUs, 14 nodes)... Connecting to Ray at 10.128.32.49:6379 (expecting 14 nodes, 56.0 GPUs) Ray connection established, polling for resources... [Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'object_store_memory': 601295421440.0, 'CPU': 4032.0, 'accelerator_type:GH200': 14.0, 'node:10.128.32.50': 1.0, 'memory': 10479941910528.0, 'GPU': 56.0, 'node:10.128.32.58': 1.0, 'node:10.128.32.60': 1.0, 'node:10.128.32.62': 1.0, 'node:10.128.32.57': 1.0, 'node:10.128.32.55': 1.0, 'node:10.128.32.51': 1.0, 'node:10.128.32.52': 1.0, 'node:10.128.32.59': 1.0, 'node:10.128.32.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.54': 1.0, 'node:10.128.32.61': 1.0, 'node:10.128.32.53': 1.0, 'node:10.128.32.56': 1.0} ✓ Ray cluster ready === Ray Cluster Ready === Address: 10.128.32.49:6379 Total GPUs: 56 ========================= Ray cluster ready at 10.128.32.49:6379 Total GPUs available: 56 [RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona) [RLJobRunner] No Pinggy tunnel needed, using local vLLM Running SkyRL: Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Entrypoint: examples.terminal_bench.entrypoints.main_tbench Args: 120 Hydra arguments Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqnorm ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqnorm trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_639476.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e 2026-06-08 20:41:23.641 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:765 - Exporting wandb api key to ray runtime env 2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:784 - Exporting RAY_ADDRESS to ray runtime env 2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:809 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0 2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:809 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET 2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:809 - Exporting `NCCL_DEBUG` to ray runtime env: WARN 2026-06-08 20:41:23,642 INFO worker.py:1680 -- Using address 10.128.32.49:6379 set in the environment variable RAY_ADDRESS 2026-06-08 20:41:23,675 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.49:6379... 2026-06-08 20:41:23,686 INFO worker.py:2007 -- Connected to Ray cluster. /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( (raylet, ip=10.128.32.54) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e 2026-06-08 20:41:25.873 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor (pid=530083) [2026-06-08 20:41:27,275 E 530083 530758] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (skyrl_entrypoint pid=546644) 2026-06-08 20:41:32.678 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:214 - SkyRL log level set to: INFO (pid=530096) [2026-06-08 20:41:29,941 E 530096 531589] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 17x across cluster] (skyrl_entrypoint pid=546644) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (skyrl_entrypoint pid=546644) 2026-06-08 20:41:32.962 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large (skyrl_entrypoint pid=546644) 2026-06-08 20:41:34.762 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories (skyrl_entrypoint pid=546644) 2026-06-08 20:41:34.763 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths (skyrl_entrypoint pid=546644) 2026-06-08 20:41:34.775 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:407 - data: (skyrl_entrypoint pid=546644) train_data: (skyrl_entrypoint pid=546644) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large (skyrl_entrypoint pid=546644) val_data: [] (skyrl_entrypoint pid=546644) trainer: (skyrl_entrypoint pid=546644) placement: (skyrl_entrypoint pid=546644) colocate_all: false (skyrl_entrypoint pid=546644) colocate_policy_ref: true (skyrl_entrypoint pid=546644) policy_num_nodes: 2 (skyrl_entrypoint pid=546644) policy_num_gpus_per_node: 4 (skyrl_entrypoint pid=546644) critic_num_nodes: 1 (skyrl_entrypoint pid=546644) critic_num_gpus_per_node: 4 (skyrl_entrypoint pid=546644) ref_num_nodes: 2 (skyrl_entrypoint pid=546644) ref_num_gpus_per_node: 4 (skyrl_entrypoint pid=546644) policy_strict_spread_pg: false (skyrl_entrypoint pid=546644) policy_per_gpu_bundles: false (skyrl_entrypoint pid=546644) policy_force_cvd_mask: false (skyrl_entrypoint pid=546644) sequence_parallel_backend: ulysses (skyrl_entrypoint pid=546644) strategy: fsdp2 (skyrl_entrypoint pid=546644) policy: (skyrl_entrypoint pid=546644) model: (skyrl_entrypoint pid=546644) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 (skyrl_entrypoint pid=546644) lora: (skyrl_entrypoint pid=546644) rank: 0 (skyrl_entrypoint pid=546644) alpha: 16 (skyrl_entrypoint pid=546644) dropout: 0 (skyrl_entrypoint pid=546644) lora_sync_path: /tmp/skyrl_lora_sync (skyrl_entrypoint pid=546644) target_modules: all-linear (skyrl_entrypoint pid=546644) exclude_modules: null (skyrl_entrypoint pid=546644) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=546644) optimizer_config: (skyrl_entrypoint pid=546644) optimizer: AdamW (skyrl_entrypoint pid=546644) lr: 8.0e-06 (skyrl_entrypoint pid=546644) adam_betas: (skyrl_entrypoint pid=546644) - 0.9 (skyrl_entrypoint pid=546644) - 0.999 (skyrl_entrypoint pid=546644) weight_decay: 0.0 (skyrl_entrypoint pid=546644) max_grad_norm: 0.9 (skyrl_entrypoint pid=546644) offload_after_step: true (skyrl_entrypoint pid=546644) num_warmup_steps: 0 (skyrl_entrypoint pid=546644) scheduler: constant_with_warmup (skyrl_entrypoint pid=546644) optimizer_kwargs: {} (skyrl_entrypoint pid=546644) fsdp_config: (skyrl_entrypoint pid=546644) cpu_offload: false (skyrl_entrypoint pid=546644) reshard_after_forward: true (skyrl_entrypoint pid=546644) fsdp_size: 4 (skyrl_entrypoint pid=546644) expert_model_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=546644) moe_token_dispatcher_type: alltoall (skyrl_entrypoint pid=546644) moe_router_replay: false (skyrl_entrypoint pid=546644) moe_grouped_gemm: false (skyrl_entrypoint pid=546644) ep_comm_backend: torch (skyrl_entrypoint pid=546644) deepep_num_sms: 20 (skyrl_entrypoint pid=546644) deepep_token_chunk_size: null (skyrl_entrypoint pid=546644) sequence_parallel_size: 1 (skyrl_entrypoint pid=546644) use_torch_compile: false (skyrl_entrypoint pid=546644) record_memory: false (skyrl_entrypoint pid=546644) megatron_config: (skyrl_entrypoint pid=546644) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=546644) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=546644) context_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_model_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_tensor_parallel_size: null (skyrl_entrypoint pid=546644) ddp_config: (skyrl_entrypoint pid=546644) grad_reduce_in_fp32: true (skyrl_entrypoint pid=546644) overlap_grad_reduce: false (skyrl_entrypoint pid=546644) overlap_param_gather: false (skyrl_entrypoint pid=546644) average_in_collective: true (skyrl_entrypoint pid=546644) model_config_kwargs: {} (skyrl_entrypoint pid=546644) torch_profiler_config: (skyrl_entrypoint pid=546644) enable: false (skyrl_entrypoint pid=546644) ranks: [] (skyrl_entrypoint pid=546644) save_path: null (skyrl_entrypoint pid=546644) optimizer_config_kwargs: (skyrl_entrypoint pid=546644) overlap_cpu_optimizer_d2h_h2d: false (skyrl_entrypoint pid=546644) use_precision_aware_optimizer: false (skyrl_entrypoint pid=546644) optimizer_cpu_offload: false (skyrl_entrypoint pid=546644) optimizer_offload_fraction: 0.0 (skyrl_entrypoint pid=546644) transformer_config_kwargs: (skyrl_entrypoint pid=546644) recompute_granularity: full (skyrl_entrypoint pid=546644) recompute_modules: (skyrl_entrypoint pid=546644) - core_attn (skyrl_entrypoint pid=546644) recompute_method: uniform (skyrl_entrypoint pid=546644) recompute_num_layers: 1 (skyrl_entrypoint pid=546644) empty_cuda_cache: true (skyrl_entrypoint pid=546644) ref: (skyrl_entrypoint pid=546644) model: (skyrl_entrypoint pid=546644) path: ${trainer.policy.model.path} (skyrl_entrypoint pid=546644) sequence_parallel_size: 1 (skyrl_entrypoint pid=546644) deepspeed_config: ${deepspeed_config.eval} (skyrl_entrypoint pid=546644) fsdp_config: (skyrl_entrypoint pid=546644) cpu_offload: false (skyrl_entrypoint pid=546644) reshard_after_forward: true (skyrl_entrypoint pid=546644) fsdp_size: 4 (skyrl_entrypoint pid=546644) expert_model_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=546644) moe_token_dispatcher_type: alltoall (skyrl_entrypoint pid=546644) moe_router_replay: false (skyrl_entrypoint pid=546644) moe_grouped_gemm: false (skyrl_entrypoint pid=546644) ep_comm_backend: torch (skyrl_entrypoint pid=546644) deepep_num_sms: 20 (skyrl_entrypoint pid=546644) deepep_token_chunk_size: null (skyrl_entrypoint pid=546644) megatron_config: (skyrl_entrypoint pid=546644) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=546644) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=546644) context_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_model_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=546644) model_config_kwargs: {} (skyrl_entrypoint pid=546644) transformer_config_kwargs: {} (skyrl_entrypoint pid=546644) critic: (skyrl_entrypoint pid=546644) model: (skyrl_entrypoint pid=546644) path: null (skyrl_entrypoint pid=546644) lora: (skyrl_entrypoint pid=546644) rank: 0 (skyrl_entrypoint pid=546644) alpha: 16 (skyrl_entrypoint pid=546644) dropout: 0 (skyrl_entrypoint pid=546644) target_modules: all-linear (skyrl_entrypoint pid=546644) exclude_modules: null (skyrl_entrypoint pid=546644) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=546644) optimizer_config: (skyrl_entrypoint pid=546644) optimizer: AdamW (skyrl_entrypoint pid=546644) lr: 5.0e-06 (skyrl_entrypoint pid=546644) adam_betas: (skyrl_entrypoint pid=546644) - 0.9 (skyrl_entrypoint pid=546644) - 0.999 (skyrl_entrypoint pid=546644) weight_decay: 0.01 (skyrl_entrypoint pid=546644) max_grad_norm: 1.0 (skyrl_entrypoint pid=546644) offload_after_step: true (skyrl_entrypoint pid=546644) num_warmup_steps: 0 (skyrl_entrypoint pid=546644) scheduler: constant_with_warmup (skyrl_entrypoint pid=546644) optimizer_kwargs: {} (skyrl_entrypoint pid=546644) fsdp_config: (skyrl_entrypoint pid=546644) cpu_offload: false (skyrl_entrypoint pid=546644) reshard_after_forward: true (skyrl_entrypoint pid=546644) fsdp_size: -1 (skyrl_entrypoint pid=546644) expert_model_parallel_size: 1 (skyrl_entrypoint pid=546644) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=546644) moe_token_dispatcher_type: alltoall (skyrl_entrypoint pid=546644) moe_router_replay: false (skyrl_entrypoint pid=546644) moe_grouped_gemm: false (skyrl_entrypoint pid=546644) ep_comm_backend: torch (skyrl_entrypoint pid=546644) deepep_num_sms: 20 (skyrl_entrypoint pid=546644) deepep_token_chunk_size: null (skyrl_entrypoint pid=546644) sequence_parallel_size: 1 (skyrl_entrypoint pid=546644) algorithm: (skyrl_entrypoint pid=546644) advantage_estimator: rloo_n (skyrl_entrypoint pid=546644) kl_ctrl: (skyrl_entrypoint pid=546644) type: fixed (skyrl_entrypoint pid=546644) kl_target: 0.1 (skyrl_entrypoint pid=546644) horizon: 10000 (skyrl_entrypoint pid=546644) kl_estimator_type: k3 (skyrl_entrypoint pid=546644) use_kl_estimator_k3: false (skyrl_entrypoint pid=546644) use_abs_kl: false (skyrl_entrypoint pid=546644) use_kl_in_reward: false (skyrl_entrypoint pid=546644) use_kl_loss: false (skyrl_entrypoint pid=546644) kl_loss_coef: 0.0 (skyrl_entrypoint pid=546644) use_entropy_loss: false (skyrl_entrypoint pid=546644) entropy_loss_coef: 0.01 (skyrl_entrypoint pid=546644) advantage_batch_normalize: false (skyrl_entrypoint pid=546644) value_head_prefix: value_head (skyrl_entrypoint pid=546644) policy_loss_type: regular (skyrl_entrypoint pid=546644) loss_reduction: seq_mean_token_sum_norm_global (skyrl_entrypoint pid=546644) global_loss_denom: null (skyrl_entrypoint pid=546644) grpo_norm_by_std: true (skyrl_entrypoint pid=546644) rloo_n_min_group_size: 4 (skyrl_entrypoint pid=546644) rloo_n_filter_zero_reward_groups: true (skyrl_entrypoint pid=546644) lambd: 1.0 (skyrl_entrypoint pid=546644) gamma: 1.0 (skyrl_entrypoint pid=546644) eps_clip_low: 0.2 (skyrl_entrypoint pid=546644) eps_clip_high: 0.05 (skyrl_entrypoint pid=546644) clip_ratio_c: 3.0 (skyrl_entrypoint pid=546644) tis_imp_ratio_cap: -1.0 (skyrl_entrypoint pid=546644) use_tis: false (skyrl_entrypoint pid=546644) sapo: (skyrl_entrypoint pid=546644) tau_pos: 1.0 (skyrl_entrypoint pid=546644) tau_neg: 1.05 (skyrl_entrypoint pid=546644) value_clip: 0.2 (skyrl_entrypoint pid=546644) dynamic_sampling: (skyrl_entrypoint pid=546644) type: null (skyrl_entrypoint pid=546644) max_sample_batches: 30 (skyrl_entrypoint pid=546644) min_replace_ratio: 0.3 (skyrl_entrypoint pid=546644) clip_cov: (skyrl_entrypoint pid=546644) clip_ratio: 0.0002 (skyrl_entrypoint pid=546644) clip_cov_lb: 1.0 (skyrl_entrypoint pid=546644) clip_cov_ub: 5.0 (skyrl_entrypoint pid=546644) kl_cov: (skyrl_entrypoint pid=546644) kl_cov_frac: 0.2 (skyrl_entrypoint pid=546644) ppo_kl_coef: 1.0 (skyrl_entrypoint pid=546644) cispo: (skyrl_entrypoint pid=546644) cispo_eps_clip_low: 0 (skyrl_entrypoint pid=546644) cispo_eps_clip_high: 5 (skyrl_entrypoint pid=546644) z_clip: (skyrl_entrypoint pid=546644) enabled: false (skyrl_entrypoint pid=546644) alpha: 0.97 (skyrl_entrypoint pid=546644) z_thresh: 2.5 (skyrl_entrypoint pid=546644) warmup_steps: 25 (skyrl_entrypoint pid=546644) mode: zscore (skyrl_entrypoint pid=546644) clip_option: adaptive_scaling (skyrl_entrypoint pid=546644) clip_factor: 1.0 (skyrl_entrypoint pid=546644) skip_update_on_spike: false (skyrl_entrypoint pid=546644) stale_clip: (skyrl_entrypoint pid=546644) enabled: false (skyrl_entrypoint pid=546644) alpha: 0.3 (skyrl_entrypoint pid=546644) entropy_threshold: 0.15 (skyrl_entrypoint pid=546644) entropy_window: 10 (skyrl_entrypoint pid=546644) min_lr_scale: 0.1 (skyrl_entrypoint pid=546644) max_seq_len: 1004095 (skyrl_entrypoint pid=546644) fully_async: (skyrl_entrypoint pid=546644) max_staleness_steps: 16 (skyrl_entrypoint pid=546644) num_parallel_generation_workers: 338 (skyrl_entrypoint pid=546644) gradient_checkpointing: true (skyrl_entrypoint pid=546644) gradient_checkpointing_use_reentrant: false (skyrl_entrypoint pid=546644) seed: 42 (skyrl_entrypoint pid=546644) resume_mode: latest (skyrl_entrypoint pid=546644) resume_path: null (skyrl_entrypoint pid=546644) ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints (skyrl_entrypoint pid=546644) max_ckpts_to_keep: -1 (skyrl_entrypoint pid=546644) ckpt_interval: 2 (skyrl_entrypoint pid=546644) hf_save_interval: 5 (skyrl_entrypoint pid=546644) hf_upload_mode: latest (skyrl_entrypoint pid=546644) export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports (skyrl_entrypoint pid=546644) bf16: true (skyrl_entrypoint pid=546644) epochs: 2 (skyrl_entrypoint pid=546644) max_steps: 80 (skyrl_entrypoint pid=546644) update_epochs_per_batch: 1 (skyrl_entrypoint pid=546644) train_batch_size: 64 (skyrl_entrypoint pid=546644) policy_mini_batch_size: 64 (skyrl_entrypoint pid=546644) critic_mini_batch_size: 256 (skyrl_entrypoint pid=546644) micro_train_batch_size_per_gpu: 1 (skyrl_entrypoint pid=546644) micro_forward_batch_size_per_gpu: 4 (skyrl_entrypoint pid=546644) update_ref_every_epoch: false (skyrl_entrypoint pid=546644) use_sample_packing: true (skyrl_entrypoint pid=546644) eval_batch_size: 64 (skyrl_entrypoint pid=546644) eval_before_train: false (skyrl_entrypoint pid=546644) eval_interval: 999999 (skyrl_entrypoint pid=546644) max_prompt_length: 999999 (skyrl_entrypoint pid=546644) flash_attn: true (skyrl_entrypoint pid=546644) disable_fast_tokenizer: false (skyrl_entrypoint pid=546644) target_modules: null (skyrl_entrypoint pid=546644) exclude_modules: null (skyrl_entrypoint pid=546644) project_name: OpenThoughts-Agent (skyrl_entrypoint pid=546644) run_name: ablation-pymethods2test-seqnorm (skyrl_entrypoint pid=546644) logger: console (skyrl_entrypoint pid=546644) tracker_commit_each_step: true (skyrl_entrypoint pid=546644) dump_data_batch: false (skyrl_entrypoint pid=546644) dump_eval_results: true (skyrl_entrypoint pid=546644) log_level: INFO (skyrl_entrypoint pid=546644) rope_scaling: null (skyrl_entrypoint pid=546644) rope_theta: null (skyrl_entrypoint pid=546644) step_wise_training: false (skyrl_entrypoint pid=546644) hf_hub_repo_id: laion/ablation-pymethods2test-seqnorm (skyrl_entrypoint pid=546644) hf_hub_private: false (skyrl_entrypoint pid=546644) hf_hub_revision: main (skyrl_entrypoint pid=546644) enable_db_registration: false (skyrl_entrypoint pid=546644) generator: (skyrl_entrypoint pid=546644) model_name: ${trainer.policy.model.path} (skyrl_entrypoint pid=546644) model_dtype: bfloat16 (skyrl_entrypoint pid=546644) timeout_multiplier: 1.0 (skyrl_entrypoint pid=546644) run_engines_locally: true (skyrl_entrypoint pid=546644) num_inference_engines: 48 (skyrl_entrypoint pid=546644) backend: vllm (skyrl_entrypoint pid=546644) weight_sync_backend: nccl (skyrl_entrypoint pid=546644) fuse_weights: false (skyrl_entrypoint pid=546644) weight_transfer_threshold_cuda_ipc_GB: 1.0 (skyrl_entrypoint pid=546644) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=546644) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=546644) inference_engine_expert_parallel_size: 1 (skyrl_entrypoint pid=546644) inference_engine_data_parallel_size: 1 (skyrl_entrypoint pid=546644) inference_engine_mp_backend: false (skyrl_entrypoint pid=546644) n_samples_per_prompt: 8 (skyrl_entrypoint pid=546644) async_engine: true (skyrl_entrypoint pid=546644) batched: false (skyrl_entrypoint pid=546644) max_input_length: ${trainer.max_prompt_length} (skyrl_entrypoint pid=546644) vllm_v1_disable_multiproc: true (skyrl_entrypoint pid=546644) enable_prefix_caching: true (skyrl_entrypoint pid=546644) enable_chunked_prefill: true (skyrl_entrypoint pid=546644) max_num_batched_tokens: 65536 (skyrl_entrypoint pid=546644) enforce_eager: true (skyrl_entrypoint pid=546644) fully_sharded_loras: false (skyrl_entrypoint pid=546644) enable_ray_prometheus_stats: false (skyrl_entrypoint pid=546644) vllm_stats_interval: 1 (skyrl_entrypoint pid=546644) gpu_memory_utilization: 0.75 (skyrl_entrypoint pid=546644) max_num_seqs: 24 (skyrl_entrypoint pid=546644) remote_inference_engine_urls: (skyrl_entrypoint pid=546644) - 127.0.0.1:8001 (skyrl_entrypoint pid=546644) enable_http_endpoint: true (skyrl_entrypoint pid=546644) http_endpoint_host: 127.0.0.1 (skyrl_entrypoint pid=546644) http_endpoint_port: 8000 (skyrl_entrypoint pid=546644) max_turns: 999999 (skyrl_entrypoint pid=546644) chat_template: (skyrl_entrypoint pid=546644) source: name (skyrl_entrypoint pid=546644) name_or_path: null (skyrl_entrypoint pid=546644) chat_template_kwargs: {} (skyrl_entrypoint pid=546644) engine_init_kwargs: (skyrl_entrypoint pid=546644) max_model_len: 32768 (skyrl_entrypoint pid=546644) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=546644) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 (skyrl_entrypoint pid=546644) override_existing_update_group: disable (skyrl_entrypoint pid=546644) sampling_params: (skyrl_entrypoint pid=546644) max_generate_length: 4096 (skyrl_entrypoint pid=546644) repetition_penalty: 1.0 (skyrl_entrypoint pid=546644) temperature: 0.7 (skyrl_entrypoint pid=546644) top_p: 0.95 (skyrl_entrypoint pid=546644) min_p: 0.0 (skyrl_entrypoint pid=546644) top_k: 20 (skyrl_entrypoint pid=546644) logprobs: null (skyrl_entrypoint pid=546644) stop: null (skyrl_entrypoint pid=546644) use_conversation_multi_turn: true (skyrl_entrypoint pid=546644) append_eos_token_after_stop_str_in_multi_turn: true (skyrl_entrypoint pid=546644) eval_sampling_params: (skyrl_entrypoint pid=546644) max_generate_length: ${generator.sampling_params.max_generate_length} (skyrl_entrypoint pid=546644) repetition_penalty: 1.0 (skyrl_entrypoint pid=546644) temperature: 0.0 (skyrl_entrypoint pid=546644) top_p: 1.0 (skyrl_entrypoint pid=546644) min_p: 0.0 (skyrl_entrypoint pid=546644) top_k: -1 (skyrl_entrypoint pid=546644) logprobs: null (skyrl_entrypoint pid=546644) stop: null (skyrl_entrypoint pid=546644) eval_n_samples_per_prompt: 8 (skyrl_entrypoint pid=546644) zero_reward_on_non_stop: false (skyrl_entrypoint pid=546644) apply_overlong_filtering: false (skyrl_entrypoint pid=546644) rope_scaling: ${trainer.rope_scaling} (skyrl_entrypoint pid=546644) rope_theta: ${trainer.rope_theta} (skyrl_entrypoint pid=546644) teacher: (skyrl_entrypoint pid=546644) model_path: null (skyrl_entrypoint pid=546644) top_k_logprobs: 256 (skyrl_entrypoint pid=546644) num_inference_engines: 1 (skyrl_entrypoint pid=546644) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=546644) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=546644) gpu_memory_utilization: 0.9 (skyrl_entrypoint pid=546644) enforce_eager: false (skyrl_entrypoint pid=546644) backend: vllm (skyrl_entrypoint pid=546644) engine_init_kwargs: {} (skyrl_entrypoint pid=546644) environment: (skyrl_entrypoint pid=546644) env_class: gsm8k (skyrl_entrypoint pid=546644) skyrl_gym: (skyrl_entrypoint pid=546644) max_env_workers: 32 (skyrl_entrypoint pid=546644) text2sql: (skyrl_entrypoint pid=546644) db_path: /home/ray/default/sql_data (skyrl_entrypoint pid=546644) llm_as_a_judge: (skyrl_entrypoint pid=546644) model: gpt-4o-mini (skyrl_entrypoint pid=546644) base_url: null (skyrl_entrypoint pid=546644) search: (skyrl_entrypoint pid=546644) log_requests: false (skyrl_entrypoint pid=546644) search_url: http://127.0.0.1:8000/retrieve (skyrl_entrypoint pid=546644) topk: 3 (skyrl_entrypoint pid=546644) timeout: 30 (skyrl_entrypoint pid=546644) rollout: (skyrl_entrypoint pid=546644) fanout: (skyrl_entrypoint pid=546644) enabled: true (skyrl_entrypoint pid=546644) num_coordinators: 4 (skyrl_entrypoint pid=546644) cpus_per_coordinator: 8 (skyrl_entrypoint pid=546644) deepspeed_config: (skyrl_entrypoint pid=546644) train: (skyrl_entrypoint pid=546644) zero_optimization: (skyrl_entrypoint pid=546644) stage: 3 (skyrl_entrypoint pid=546644) offload_param: (skyrl_entrypoint pid=546644) device: none (skyrl_entrypoint pid=546644) offload_optimizer: (skyrl_entrypoint pid=546644) device: none (skyrl_entrypoint pid=546644) pin_memory: true (skyrl_entrypoint pid=546644) sub_group_size: auto (skyrl_entrypoint pid=546644) reduce_bucket_size: auto (skyrl_entrypoint pid=546644) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=546644) stage3_prefetch_bucket_size: auto (skyrl_entrypoint pid=546644) stage3_max_live_parameters: auto (skyrl_entrypoint pid=546644) stage3_max_reuse_distance: auto (skyrl_entrypoint pid=546644) round_robin_gradients: true (skyrl_entrypoint pid=546644) zero_hpz_partition_size: 1 (skyrl_entrypoint pid=546644) zero_quantized_weights: false (skyrl_entrypoint pid=546644) zero_quantized_gradients: false (skyrl_entrypoint pid=546644) torch_autocast: (skyrl_entrypoint pid=546644) enabled: true (skyrl_entrypoint pid=546644) dtype: bfloat16 (skyrl_entrypoint pid=546644) disable_trace_cache: false (skyrl_entrypoint pid=546644) data_types: (skyrl_entrypoint pid=546644) grad_accum_dtype: fp32 (skyrl_entrypoint pid=546644) gradient_clipping: 1.0 (skyrl_entrypoint pid=546644) wall_clock_breakdown: false (skyrl_entrypoint pid=546644) prescale_gradient: false (skyrl_entrypoint pid=546644) eval: (skyrl_entrypoint pid=546644) zero_optimization: (skyrl_entrypoint pid=546644) stage: 3 (skyrl_entrypoint pid=546644) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=546644) offload_param: (skyrl_entrypoint pid=546644) device: cpu (skyrl_entrypoint pid=546644) pin_memory: true (skyrl_entrypoint pid=546644) torch_autocast: (skyrl_entrypoint pid=546644) enabled: true (skyrl_entrypoint pid=546644) dtype: bfloat16 (skyrl_entrypoint pid=546644) gradient_clipping: 1.0 (skyrl_entrypoint pid=546644) prescale_gradient: false (skyrl_entrypoint pid=546644) wall_clock_breakdown: false (skyrl_entrypoint pid=546644) terminal_bench_config: (skyrl_entrypoint pid=546644) trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/trace_jobs (skyrl_entrypoint pid=546644) harbor: (skyrl_entrypoint pid=546644) name: terminus-2 (skyrl_entrypoint pid=546644) max_episodes: 999999 (skyrl_entrypoint pid=546644) enable_summarize: false (skyrl_entrypoint pid=546644) store_all_messages: true (skyrl_entrypoint pid=546644) trajectory_config: (skyrl_entrypoint pid=546644) raw_content: true (skyrl_entrypoint pid=546644) enable_episode_logging: false (skyrl_entrypoint pid=546644) record_terminal_session: false (skyrl_entrypoint pid=546644) enable_pane_logging: false (skyrl_entrypoint pid=546644) strict_json_parser: true (skyrl_entrypoint pid=546644) interleaved_thinking: true (skyrl_entrypoint pid=546644) extra_body: (skyrl_entrypoint pid=546644) chat_template_kwargs: (skyrl_entrypoint pid=546644) enable_thinking: true (skyrl_entrypoint pid=546644) override_timeout_sec: 900 (skyrl_entrypoint pid=546644) override_cpus: 1 (skyrl_entrypoint pid=546644) override_memory_mb: 2048 (skyrl_entrypoint pid=546644) override_storage_mb: 2048 (skyrl_entrypoint pid=546644) auto_snapshot: true (skyrl_entrypoint pid=546644) verifier_override_timeout_sec: 120 (skyrl_entrypoint pid=546644) max_retries: 3 (skyrl_entrypoint pid=546644) min_wait_sec: 60.0 (skyrl_entrypoint pid=546644) max_wait_sec: 600.0 (skyrl_entrypoint pid=546644) wait_multiplier: 2.0 (skyrl_entrypoint pid=546644) exclude_exceptions: (skyrl_entrypoint pid=546644) - VerifierTimeoutError (skyrl_entrypoint pid=546644) - VerifierRuntimeError (skyrl_entrypoint pid=546644) - RewardFileNotFoundError (skyrl_entrypoint pid=546644) - RewardFileEmptyError (skyrl_entrypoint pid=546644) - VerifierOutputParseError (skyrl_entrypoint pid=546644) n_concurrent_trials: 675 (skyrl_entrypoint pid=546644) log_level: INFO (skyrl_entrypoint pid=546644) enable_reward_shaping: false (skyrl_entrypoint pid=546644) enable_error_classification: true (skyrl_entrypoint pid=546644) mask_exceptions: (skyrl_entrypoint pid=546644) - DaytonaError (skyrl_entrypoint pid=546644) - EnvironmentStartTimeoutError (skyrl_entrypoint pid=546644) - NetworkError (skyrl_entrypoint pid=546644) - ConnectionError (skyrl_entrypoint pid=546644) - RewardFileNotFoundError (skyrl_entrypoint pid=546644) - RewardFileEmptyError (skyrl_entrypoint pid=546644) - AgentEnvironmentTimeoutError (skyrl_entrypoint pid=546644) - ContextLengthExceededError (skyrl_entrypoint pid=546644) default_error_treatment: zero (skyrl_entrypoint pid=546644) passthrough_exceptions: (skyrl_entrypoint pid=546644) - AgentTimeoutError (skyrl_entrypoint pid=546644) zero_exceptions: [] (skyrl_entrypoint pid=546644) model_info: (skyrl_entrypoint pid=546644) max_input_tokens: 32000 (skyrl_entrypoint pid=546644) max_output_tokens: 4096 (skyrl_entrypoint pid=546644) archiving: (skyrl_entrypoint pid=546644) enabled: false (skyrl_entrypoint pid=546644) trace_upload: (skyrl_entrypoint pid=546644) enabled: true (skyrl_entrypoint pid=546644) repo_org: DCAgent (skyrl_entrypoint pid=546644) episodes: last (skyrl_entrypoint pid=546644) dataset_type: SFT (skyrl_entrypoint pid=546644) cleanup: true (skyrl_entrypoint pid=546644)  (skyrl_entrypoint pid=546644) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (skyrl_entrypoint pid=546644) No module named 'vllm._version' (skyrl_entrypoint pid=546644) from .version import __version__, __version_tuple__ # isort:skip (skyrl_entrypoint pid=546644) W0608 20:41:39.902000 546644 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' (pid=530352) [2026-06-08 20:41:37,547 E 530352 544574] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster] (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (raylet, ip=10.128.32.57) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 43x across cluster] (pid=2689828, ip=10.128.32.54) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (pid=2689828, ip=10.128.32.54) No module named 'vllm._version' (pid=2689828, ip=10.128.32.54) from .version import __version__, __version_tuple__ # isort:skip (raylet, ip=10.128.32.58) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 109x across cluster] [2026-06-08 20:41:53,824 E 546292 546616] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (pid=321222, ip=10.128.32.56) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster] (pid=321222, ip=10.128.32.56) No module named 'vllm._version' [repeated 15x across cluster] (pid=321222, ip=10.128.32.56) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster] (RegistryActor pid=2689760, ip=10.128.32.54) [2026-06-08 20:41:54,703 E 2689760 2689800] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.32.55) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 71x across cluster] (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:41:59.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING= (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:41:59.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:41:59.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity (pid=2794792, ip=10.128.32.52) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 9x across cluster] (pid=2794792, ip=10.128.32.52) No module named 'vllm._version' [repeated 9x across cluster] (pid=2794792, ip=10.128.32.52) from .version import __version__, __version_tuple__ # isort:skip [repeated 9x across cluster] (skyrl_entrypoint pid=546644) [2026-06-08 20:41:56,879 E 546644 546687] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:42:00.144 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0 (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:42:00.168 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:42:00.168 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:42:00.168 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:42:00.168 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) 2026-06-08 20:42:00.190 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.00s (attempt 1/5) to avoid port collisions (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (raylet, ip=10.128.32.61) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 128x across cluster] (skyrl_entrypoint pid=546644) [2026-06-08 20:42:04] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000 (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:03.798 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:03.799 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:03.799 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54) No module named 'vllm._version' [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54) from .version import __version__, __version_tuple__ # isort:skip [repeated 19x across cluster] (skyrl_entrypoint pid=546644) [2026-06-08 20:42:04] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint... (skyrl_entrypoint pid=546644) [2026-06-08 20:42:05] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds) (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.367 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000 (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.367 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines. (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.369 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.370 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.371 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.372 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156 (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.372 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000 (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.372 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78 (skyrl_entrypoint pid=546644) 2026-06-08 20:42:05.372 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80 (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:05.012 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:05.034 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:05.034 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:05.034 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:05.034 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) 2026-06-08 20:42:05.061 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 1.75s (attempt 1/5) to avoid port collisions [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 45x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) 2026-06-08 20:42:09.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) 2026-06-08 20:42:09.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) 2026-06-08 20:42:09.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) No module named 'vllm._version' [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) from .version import __version__, __version_tuple__ # isort:skip [repeated 18x across cluster] (pid=3886563, ip=10.128.32.62) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) 2026-06-08 20:42:07.994 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) 2026-06-08 20:42:08.016 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) 2026-06-08 20:42:08.016 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) 2026-06-08 20:42:08.017 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) 2026-06-08 20:42:08.017 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) 2026-06-08 20:42:08.046 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.17s (attempt 1/5) to avoid port collisions [repeated 8x across cluster] (bundle_reservation_check_func pid=546719) [2026-06-08 20:42:12,634 E 546719 546759] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:10.747 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:10.748 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:10.748 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55) No module named 'vllm._version' [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55) from .version import __version__, __version_tuple__ # isort:skip [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:12.143 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:12.166 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:12.166 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:12.167 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:12.167 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) 2026-06-08 20:42:12.200 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.09s (attempt 1/5) to avoid port collisions [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53) (EngineCore_DP0 pid=2730194) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) (AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53) (EngineCore_DP0 pid=2730194) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53) [2026-06-08 20:42:16,474 E 2729834 2729931] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2729961, ip=10.128.32.53) (EngineCore_DP0 pid=2730198) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) (EngineCore_DP0 pid=2690185) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54) (EngineCore_DP0 pid=2690206) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) 2026-06-08 20:42:18.145 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) 2026-06-08 20:42:18.165 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) 2026-06-08 20:42:18.165 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60) No module named 'vllm._version' [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54) (EngineCore_DP0 pid=2690189) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) (EngineCore_DP0 pid=3870167) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) (EngineCore_DP0 pid=2730214) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3869809, ip=10.128.32.57) (EngineCore_DP0 pid=3870171) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) [W608 20:42:20.803504224 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-06.jupiter.internal]:38653 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) [W608 20:42:20.805060772 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) [rank0]:[W608 20:42:20.810164201 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) (AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54) (EngineCore_DP0 pid=2690201) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53) (EngineCore_DP0 pid=2730190) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) (EngineCore_DP0 pid=321457) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) 2026-06-08 20:42:19.386 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61) 2026-06-08 20:42:19.409 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61) 2026-06-08 20:42:19.409 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61) 2026-06-08 20:42:19.409 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61) 2026-06-08 20:42:19.409 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) 2026-06-08 20:42:19.436 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.66s (attempt 1/5) to avoid port collisions [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56) (EngineCore_DP0 pid=321462) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) (EngineCore_DP0 pid=3870181) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=321221, ip=10.128.32.56) (EngineCore_DP0 pid=321449) _C._set_float32_matmul_precision(precision) (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:42:21 INFO [ipv4-debug] hostname=jpbo-042-14.jupiter.internal (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:42:21 INFO [ipv4-debug] _global_node.node_ip_address=10.128.32.62 (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:42:21 INFO [ipv4-debug] get_node_ip_address()=10.128.32.62 (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) (EngineCore_DP0 pid=3948025) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) (EngineCore_DP0 pid=3870176) _C._set_float32_matmul_precision(precision) (skyrl_entrypoint pid=546644) 2026-06-08 20:42:22.012 | INFO  | skyrl_train.workers.worker:_initiate_actors:636 - Initializing process group for RayActorGroup (AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54) (EngineCore_DP0 pid=2690189) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00 10.128.32.49 (routable head IP) for coordinator litellm base_url connectivity (skyrl_entrypoint pid=546644) 2026-06-08 20:42:48.824 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:405 - [RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8 (skyrl_entrypoint pid=546644) 2026-06-08 20:42:48.829 | INFO  | examples.terminal_bench.rollout_coordinator:startup:434 - [RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD) (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) (EngineCore_DP0 pid=3893429) (AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61) (EngineCore_DP0 pid=3893421) (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) (EngineCore_DP0 pid=3893442) (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) (AsyncVLLMInferenceEngine pid=4003580, ip=10.128.32.60) (EngineCore_DP0 pid=4003942) (AsyncVLLMInferenceEngine pid=4003710, ip=10.128.32.60) (EngineCore_DP0 pid=4003961) (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) (EngineCore_DP0 pid=4003956) (AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60) (EngineCore_DP0 pid=4003938) (AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51) (EngineCore_DP0 pid=2916423) (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) (EngineCore_DP0 pid=2916417) (AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51) (EngineCore_DP0 pid=2916413) (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) (EngineCore_DP0 pid=3893429) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 13x across cluster] (FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62) [rank1]:[W608 20:42:46.956187698 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) 2026-06-08 20:42:49.702 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.45s/it] [repeated 36x across cluster] (FSDPPolicyWorkerBase pid=546825) [2026-06-08 20:42:52,793 E 546825 546973] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.371 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster] (skyrl_entrypoint pid=546644) 2026-06-08 20:42:54.609 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 1/4 started (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.32.49:8000) (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.608 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.608 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=547199) 2026-06-08 20:42:54.608 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete (RolloutCoordinator pid=547199) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51) 2026-06-08 20:42:53.595 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 12x across cluster] (FSDPPolicyWorkerBase pid=546824) [2026-06-08 20:42:52,896 E 546824 547051] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:01.985 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (raylet, ip=10.128.32.56) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.32.49:8000) (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.205 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.205 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=321702, ip=10.128.32.56) 2026-06-08 20:43:02.205 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete (skyrl_entrypoint pid=546644) 2026-06-08 20:43:02.205 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 2/4 started (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.597 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (RolloutCoordinator pid=321702, ip=10.128.32.56) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (raylet, ip=10.128.32.54) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster] (skyrl_entrypoint pid=546644) 2026-06-08 20:43:10.821 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 3/4 started (RolloutCoordinator pid=2690439, ip=10.128.32.54) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.817 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.817 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.817 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.818 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.32.49:8000) (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.820 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.820 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=2690439, ip=10.128.32.54) 2026-06-08 20:43:10.820 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.562 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (raylet, ip=10.128.32.50) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster] (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.782 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 4/4 started (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.782 | INFO  | examples.terminal_bench.rollout_coordinator:startup:477 - [RolloutDispatcher] 4 coordinators started (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.782 | INFO  | skyrl_train.fully_async_trainer:train:485 - Generator startup complete (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.782 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Started: 'load_checkpoints' (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.788 | INFO  | skyrl_train.trainer:load_checkpoints:1735 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83 (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.788 | INFO  | skyrl_train.trainer:load_checkpoints:1741 - Resuming from global_step: 83 (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.794 | INFO  | skyrl_train.trainer:load_checkpoints:1757 - Successfully loaded trainer state (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.796 | INFO  | skyrl_train.trainer:load_checkpoints:1767 - Successfully loaded dataloader state (skyrl_entrypoint pid=546644) 2026-06-08 20:43:18.796 | INFO  | skyrl_train.trainer:load_checkpoints:1776 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy (RolloutCoordinator pid=3120643, ip=10.128.32.50) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.32.49:8000) (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.781 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.781 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=3120643, ip=10.128.32.50) 2026-06-08 20:43:18.781 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete (RolloutCoordinator pid=547199) [2026-06-08 20:43:19,778 E 547199 547263] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RolloutCoordinator pid=321702, ip=10.128.32.56) [2026-06-08 20:43:27,452 E 321702 321742] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RolloutCoordinator pid=2690439, ip=10.128.32.54) [2026-06-08 20:43:35,103 E 2690439 2690479] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) warnings.warn( # warn only once (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.246 | INFO  | skyrl_train.trainer:load_checkpoints:1786 - Successfully loaded policy checkpoint (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.246 | INFO  | skyrl_train.trainer:load_checkpoints:1802 - Successfully loaded complete checkpoint state from global_step_83 (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.247 | INFO  | skyrl_train.fully_async_trainer:_train_loop:512 - Resumed training from global_step 83 (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.247 | WARNING  | skyrl_train.fully_async_trainer:_train_loop:517 - No data consumption state found in checkpoint — resume may re-train on already-consumed data (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.248 | WARNING  | skyrl_train.fully_async_trainer:_train_loop:537 - Data consumption count mismatch on resume: expected 320, got 0. This can happen after epoch boundary transitions or error recovery. (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.248 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Finished: 'load_checkpoints', time cost: 18.47s (skyrl_entrypoint pid=546644) 2026-06-08 20:43:37.248 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Started: 'init_weight_sync_state' (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) No module named 'vllm._version' (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) from .version import __version__, __version_tuple__ # isort:skip (RolloutCoordinator pid=3120643, ip=10.128.32.50) [2026-06-08 20:43:43,749 E 3120643 3120685] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (FSDPPolicyWorkerBase pid=546824) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=546824) No module named 'vllm._version' [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=546824) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:43:43.980 | INFO  | logging:info:2216 - [weight-sync] hostname=jpbo-042-14.jupiter.internal (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:43:43.980 | INFO  | logging:info:2216 - [weight-sync] _global_node.node_ip_address=10.128.32.62 (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:43:43.981 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.32.62 (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) 2026-06-08 20:43:43.981 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.32.62, master_port=56205 (AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51) (EngineCore_DP0 pid=2916423) 2026-06-08 20:43:44.137 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:223 - torch.distributed.get_rank(): 0, rank_offset: 33, rank: 33, world_size: 49, group_name: skyrl (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) [rank0]:[W608 20:43:44.487817648 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-14.jupiter.internal]:56205 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51) (EngineCore_DP0 pid=2916413) 2026-06-08 20:43:44.140 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:234 - init_weight_update_communicator: master_address=10.128.32.62, master_port=56205, (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) warnings.warn( # warn only once (skyrl_entrypoint pid=546644) 2026-06-08 20:43:44.211 | INFO  | skyrl_train.trainer:init_weight_sync_state:874 - Initialized weight sync state for policy model and inference engines. (skyrl_entrypoint pid=546644) 2026-06-08 20:43:44.212 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Finished: 'init_weight_sync_state', time cost: 6.96s (AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56) [rank0]:[W608 20:43:47.408195056 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) (skyrl_entrypoint pid=546644) 2026-06-08 20:43:52.780 | INFO  | skyrl_train.fully_async_trainer:_train_loop:548 - Finished: 'sync_weights_to_inference_engines', time cost: 8.57s (skyrl_entrypoint pid=546644) 2026-06-08 20:43:52.781 | INFO  | skyrl_train.callbacks.builtin:on_train_begin:270 - HFHubUploadCallback initialized: repo=laion/ablation-pymethods2test-seqnorm, upload_steps=5, export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports (skyrl_entrypoint pid=546644) Training Step Progress: 83it [00:00, ?it/s] (skyrl_entrypoint pid=546644) Generation Buffer Progress: 0%| | 0/64 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) WARNING 06-08 20:42:10 [arg_utils.py:1256] The global random seed is set to 60. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:42:10 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:42:10 [model.py:1549] Using max model len 32768 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:42:10 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:42:10 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:42:10 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) WARNING 06-08 20:42:10 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:42:10 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) WARNING 06-08 20:42:13 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) WARNING 06-08 20:42:13 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster] (pid=4003708, ip=10.128.32.60) INFO 06-08 20:42:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) (EngineCore_DP0 pid=3870176) INFO 06-08 20:42:17 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=48, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) WARNING 06-08 20:42:15 [arg_utils.py:1256] The global random seed is set to 79. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:42:15 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:42:15 [model.py:1549] Using max model len 32768 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:42:15 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:42:15 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:42:15 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) WARNING 06-08 20:42:15 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:42:15 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53) (EngineCore_DP0 pid=2730194) INFO 06-08 20:42:17 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) WARNING 06-08 20:42:15 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) WARNING 06-08 20:42:15 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) INFO 06-08 20:42:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 25x across cluster] (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:20 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.54:38653 backend=nccl (AsyncVLLMInferenceEngine pid=2729961, ip=10.128.32.53) (EngineCore_DP0 pid=2730198) INFO 06-08 20:42:20 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:20 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... (pid=3886563, ip=10.128.32.62) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54) (EngineCore_DP0 pid=2690189) INFO 06-08 20:42:22 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:21 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=68, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) WARNING 06-08 20:42:22 [arg_utils.py:1256] The global random seed is set to 87. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:42:22 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:42:22 [model.py:1549] Using max model len 32768 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:42:22 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:42:22 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:42:22 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) WARNING 06-08 20:42:22 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:42:22 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) (EngineCore_DP0 pid=3948034) INFO 06-08 20:42:22 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 17x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) WARNING 06-08 20:42:22 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) WARNING 06-08 20:42:22 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:42:23 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58) (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:24 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.58:42977 backend=nccl [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58) (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:24 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58) (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:24 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58) (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:25 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) (EngineCore_DP0 pid=3120402) INFO 06-08 20:42:24 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=62, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:27 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 10x across cluster] (pid=3886639, ip=10.128.32.62) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=768604, ip=10.128.32.59) INFO 06-08 20:42:29 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:29 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.50:36953 backend=nccl [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:29 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:30 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:31 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:31 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=89, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) (EngineCore_DP0 pid=3893429) INFO 06-08 20:42:33 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 12x across cluster] (pid=546824) ⚙️ Running in WANDB offline mode [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) INFO 06-08 20:42:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.61:59347 backend=nccl [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:36 [default_loader.py:293] Loading weights took 14.13 seconds (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:36 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.646981 seconds (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) (EngineCore_DP0 pid=321457) INFO 06-08 20:42:39 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) (EngineCore_DP0 pid=321457) INFO 06-08 20:42:39 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) (EngineCore_DP0 pid=321457) INFO 06-08 20:42:39 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) (EngineCore_DP0 pid=2730214) INFO 06-08 20:42:40 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=3869809, ip=10.128.32.57) (EngineCore_DP0 pid=3870171) INFO 06-08 20:42:40 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.33 seconds (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) (EngineCore_DP0 pid=3870176) WARNING 06-08 20:42:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) (EngineCore_DP0 pid=3870176) INFO 06-08 20:42:40 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) (EngineCore_DP0 pid=3870176) WARNING 06-08 20:42:40 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) (EngineCore_DP0 pid=3870176) INFO 06-08 20:42:40 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54) WARNING 06-08 20:42:40 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55) (EngineCore_DP0 pid=2999262) INFO 06-08 20:42:41 [default_loader.py:293] Loading weights took 13.38 seconds [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55) (EngineCore_DP0 pid=2999262) INFO 06-08 20:42:41 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.428531 seconds [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55) (EngineCore_DP0 pid=2999262) INFO 06-08 20:42:44 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) WARNING 06-08 20:42:45 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:45 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) WARNING 06-08 20:42:45 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:45 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) WARNING 06-08 20:42:45 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) NCCL version 2.27.7+cuda13.0 (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:45 [default_loader.py:293] Loading weights took 14.23 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:46 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.346670 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:48 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:48 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:48 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:49 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.84 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) WARNING 06-08 20:42:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:49 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) WARNING 06-08 20:42:49 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:49 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) WARNING 06-08 20:42:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:49 [default_loader.py:293] Loading weights took 13.80 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:50 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.929588 seconds [repeated 12x across cluster] (RolloutCoordinator pid=547199) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=547199) [fd-monitor] [20:42:54] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:42:54] OK: RSS 0.78 GiB | node mem 235.5/858.0 GiB used (27.5%), avail 622.4 GiB (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) (EngineCore_DP0 pid=2916417) INFO 06-08 20:42:52 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) (EngineCore_DP0 pid=2916417) INFO 06-08 20:42:52 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) (EngineCore_DP0 pid=2916417) INFO 06-08 20:42:52 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51) (EngineCore_DP0 pid=2916413) INFO 06-08 20:42:52 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:52 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) WARNING 06-08 20:42:53 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:53 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) WARNING 06-08 20:42:53 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51) (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:53 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51) WARNING 06-08 20:42:53 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:43:01] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:43:01] OK: RSS 0.78 GiB | node mem 399.2/858.0 GiB used (46.5%), avail 458.7 GiB (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:43:10] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:43:10] OK: RSS 0.78 GiB | node mem 377.6/858.0 GiB used (44.0%), avail 480.4 GiB (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:43:18] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:43:18] OK: RSS 0.78 GiB | node mem 415.1/858.0 GiB used (48.4%), avail 442.9 GiB (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy/model_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy/extra_state_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy/optim_world_size_8_rank_0.pt (skyrl_entrypoint pid=546644) [fd-monitor] [20:43:32] OK: 102 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:43:32] OK: RSS 1.64 GiB | node mem 271.4/858.0 GiB used (31.6%), avail 586.6 GiB (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Successfully loaded model state dict (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Successfully loaded optimizer state (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Successfully loaded scheduler state (FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy (FSDPPolicyWorkerBase pid=546822) INFO 06-08 20:43:40 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s (AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53) (EngineCore_DP0 pid=2730190) INFO 06-08 20:43:52 [block_pool.py:452] Successfully reset prefix cache (FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62) INFO 06-08 20:43:42 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 7x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) (EngineCore_DP0 pid=3120407) INFO 06-08 20:43:52 [block_pool.py:452] Successfully reset prefix cache [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 1512x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 503x across cluster] (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) INFO 06-08 20:44:42 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) INFO 06-08 20:44:42 [loggers.py:259] Engine 000: Avg prompt throughput: 8.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57) INFO 06-08 20:44:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 45.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0% (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) INFO 06-08 20:44:48 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) INFO 06-08 20:44:48 [loggers.py:259] Engine 000: Avg prompt throughput: 8.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (AsyncVLLMInferenceEngine pid=4003580, ip=10.128.32.60) INFO 06-08 20:44:52 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) INFO 06-08 20:44:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% [repeated 4x across cluster] (RolloutCoordinator pid=547199) [fd-monitor] [20:44:54] OK: 215 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:44:54] OK: RSS 1.07 GiB | node mem 256.5/858.0 GiB used (29.9%), avail 601.5 GiB (AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54) INFO 06-08 20:44:57 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) INFO 06-08 20:44:58 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0% [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2794792, ip=10.128.32.52) INFO 06-08 20:44:59 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:45:01] OK: 330 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:45:01] OK: RSS 1.07 GiB | node mem 407.9/858.0 GiB used (47.5%), avail 450.1 GiB (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) INFO 06-08 20:45:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) INFO 06-08 20:45:04 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:45:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% [repeated 9x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:45:10] OK: 393 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:45:10] OK: RSS 1.11 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.7 GiB (AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57) INFO 06-08 20:45:10 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51) INFO 06-08 20:45:13 [loggers.py:259] Engine 000: Avg prompt throughput: 8.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=768605, ip=10.128.32.59) INFO 06-08 20:45:15 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:45:18] OK: 403 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:45:18] OK: RSS 1.12 GiB | node mem 423.5/858.0 GiB used (49.4%), avail 434.4 GiB (AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51) INFO 06-08 20:45:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 12x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54) INFO 06-08 20:45:19 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:45:22 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51) INFO 06-08 20:45:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 12x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:45:26 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51) INFO 06-08 20:45:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0% [repeated 12x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 2x across cluster] (skyrl_entrypoint pid=546644) [fd-monitor] [20:45:32] OK: 135 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:45:32] OK: RSS 1.75 GiB | node mem 256.6/858.0 GiB used (29.9%), avail 601.4 GiB (AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51) INFO 06-08 20:45:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0% [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54) INFO 06-08 20:45:39 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 14x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:45:45 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 53.2% [repeated 14x across cluster] (AsyncVLLMInferenceEngine pid=768604, ip=10.128.32.59) INFO 06-08 20:45:49 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:45:51 [loggers.py:259] Engine 000: Avg prompt throughput: 153.9 tokens/s, Avg generation throughput: 74.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 56.2% [repeated 17x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2794924, ip=10.128.32.52) INFO 06-08 20:45:53 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:45:56 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 86.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 56.2% [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58) INFO 06-08 20:45:59 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:46:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 86.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 56.2% [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=4003710, ip=10.128.32.60) INFO 06-08 20:46:04 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:46:06 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 56.2% [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52) INFO 06-08 20:46:06 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:46:11 [loggers.py:259] Engine 000: Avg prompt throughput: 76.6 tokens/s, Avg generation throughput: 55.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 57.2% [repeated 27x across cluster] (AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55) INFO 06-08 20:46:11 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54) INFO 06-08 20:46:15 [loggers.py:259] Engine 000: Avg prompt throughput: 24.7 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 32.8% [repeated 27x across cluster] (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:46:16 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:46:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 225.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 66.7% [repeated 31x across cluster] (AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59) INFO 06-08 20:46:21 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50) INFO 06-08 20:46:26 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 191.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 66.7% [repeated 31x across cluster] (AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53) INFO 06-08 20:46:29 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:46:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 36x across cluster] (AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61) INFO 06-08 20:46:32 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:46:36 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0% [repeated 36x across cluster] (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:46:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0% [repeated 33x across cluster] (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:46:46 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0% [repeated 35x across cluster] (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:46:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0% [repeated 37x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55) INFO 06-08 20:46:53 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) [fd-monitor] [20:46:54] OK: 256 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:46:54] OK: RSS 1.11 GiB | node mem 256.6/858.0 GiB used (29.9%), avail 601.4 GiB (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:46:56 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0% [repeated 38x across cluster] (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60) INFO 06-08 20:46:55 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:47:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% [repeated 36x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:47:01] OK: 260 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:47:01] OK: RSS 1.12 GiB | node mem 408.0/858.0 GiB used (47.5%), avail 450.0 GiB (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) INFO 06-08 20:46:59 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:47:06 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0% [repeated 38x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 3x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:47:10] OK: 388 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:47:10] OK: RSS 1.13 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.7 GiB (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:47:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0% [repeated 38x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60) INFO 06-08 20:47:16 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0% [repeated 36x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:47:18] OK: 404 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:47:18] OK: RSS 1.13 GiB | node mem 423.8/858.0 GiB used (49.4%), avail 434.2 GiB (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55) INFO 06-08 20:47:21 [loggers.py:259] Engine 000: Avg prompt throughput: 88.4 tokens/s, Avg generation throughput: 30.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 76.0% [repeated 39x across cluster] (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59) INFO 06-08 20:47:26 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 188.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 69.0% [repeated 43x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:47:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 46.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 63.1% [repeated 38x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (skyrl_entrypoint pid=546644) [fd-monitor] [20:47:32] OK: 228 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:47:32] OK: RSS 1.76 GiB | node mem 256.6/858.0 GiB used (29.9%), avail 601.4 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:47:36 [loggers.py:259] Engine 000: Avg prompt throughput: 153.0 tokens/s, Avg generation throughput: 82.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 67.6% [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:47:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 92.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 67.6% [repeated 38x across cluster] (AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58) INFO 06-08 20:47:43 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:47:46 [loggers.py:259] Engine 000: Avg prompt throughput: 75.9 tokens/s, Avg generation throughput: 70.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 70.5% [repeated 37x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:47:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 91.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 70.5% [repeated 39x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50) INFO 06-08 20:47:54 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:47:56 [loggers.py:259] Engine 000: Avg prompt throughput: 140.2 tokens/s, Avg generation throughput: 108.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 73.9% [repeated 42x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 134.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 73.9% [repeated 42x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 3x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:06 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 90.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 73.9% [repeated 41x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 7x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 90.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 73.9% [repeated 45x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:48:16 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50) INFO 06-08 20:48:16 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 84.6% [repeated 44x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 20x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:21 [loggers.py:259] Engine 000: Avg prompt throughput: 119.5 tokens/s, Avg generation throughput: 166.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 75.5% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:26 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 136.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 75.5% [repeated 42x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 6x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 129.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 75.5% [repeated 40x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 27x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:37 [loggers.py:259] Engine 000: Avg prompt throughput: 76.6 tokens/s, Avg generation throughput: 170.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 78.4% [repeated 41x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56) INFO 06-08 20:48:40 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 78.4% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:47 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 78.4% [repeated 45x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:52 [loggers.py:259] Engine 000: Avg prompt throughput: 72.9 tokens/s, Avg generation throughput: 199.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 78.5% [repeated 46x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) [fd-monitor] [20:48:54] OK: 284 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:48:54] OK: RSS 1.14 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 16x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:48:57 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 78.5% [repeated 45x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 19x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:49:01] OK: 383 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:49:01] OK: RSS 1.16 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 450.0 GiB (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 138.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 78.5% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:07 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 116.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 78.5% [repeated 45x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 8x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:49:10] OK: 345 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:49:10] OK: RSS 1.14 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.7 GiB (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54) INFO 06-08 20:49:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 138.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 84.1% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 46x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:17 [loggers.py:259] Engine 000: Avg prompt throughput: 67.2 tokens/s, Avg generation throughput: 122.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 78.8% [repeated 45x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:49:18] OK: 264 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:49:18] OK: RSS 1.14 GiB | node mem 423.8/858.0 GiB used (49.4%), avail 434.1 GiB (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 8x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 96.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 78.8% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:27 [loggers.py:259] Engine 000: Avg prompt throughput: 113.4 tokens/s, Avg generation throughput: 63.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 78.9% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:32 [loggers.py:259] Engine 000: Avg prompt throughput: 33.9 tokens/s, Avg generation throughput: 97.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 79.8% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (skyrl_entrypoint pid=546644) [fd-monitor] [20:49:32] OK: 287 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:49:32] OK: RSS 1.77 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 21x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 79.8% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:42 [loggers.py:259] Engine 000: Avg prompt throughput: 22.7 tokens/s, Avg generation throughput: 50.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 80.8% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58) INFO 06-08 20:49:47 [loggers.py:259] Engine 000: Avg prompt throughput: 210.5 tokens/s, Avg generation throughput: 117.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 82.3% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:49:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 75.6% [repeated 46x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 6x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:49:57 [loggers.py:259] Engine 000: Avg prompt throughput: 122.0 tokens/s, Avg generation throughput: 50.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 80.2% [repeated 46x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 14x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 80.2% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:07 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 80.2% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 22x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:12 [loggers.py:259] Engine 000: Avg prompt throughput: 23.2 tokens/s, Avg generation throughput: 77.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 85.4% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56) INFO 06-08 20:50:13 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 14x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:17 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 94.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 85.4% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 94.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 85.4% [repeated 46x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 19x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:27 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 85.4% [repeated 44x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 23x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:32 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 85.4% [repeated 45x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 25x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 85.4% [repeated 46x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 17x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 85.4% [repeated 46x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:47 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 85.4% [repeated 46x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 7x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57) INFO 06-08 20:50:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 85.4% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) [fd-monitor] [20:50:54] OK: 273 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:50:54] OK: RSS 1.15 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:50:57 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 123.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 85.8% [repeated 46x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 11x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:51:01] OK: 399 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:51:01] OK: RSS 1.16 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 450.0 GiB (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 92.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 85.8% [repeated 45x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:07 [loggers.py:259] Engine 000: Avg prompt throughput: 136.6 tokens/s, Avg generation throughput: 135.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 85.7% [repeated 45x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:51:10] OK: 361 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:51:10] OK: RSS 1.15 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.6 GiB (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:12 [loggers.py:259] Engine 000: Avg prompt throughput: 42.8 tokens/s, Avg generation throughput: 164.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.4%, Prefix cache hit rate: 85.9% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:17 [loggers.py:259] Engine 000: Avg prompt throughput: 99.3 tokens/s, Avg generation throughput: 216.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 85.5% [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:51:18] OK: 276 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:51:18] OK: RSS 1.15 GiB | node mem 423.9/858.0 GiB used (49.4%), avail 434.1 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 206.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.4%, Prefix cache hit rate: 85.5% [repeated 49x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:27 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 85.5% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (skyrl_entrypoint pid=546644) [fd-monitor] [20:51:32] OK: 332 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:51:32] OK: RSS 1.78 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.2 GiB (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:32 [loggers.py:259] Engine 000: Avg prompt throughput: 92.1 tokens/s, Avg generation throughput: 219.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.1%, Prefix cache hit rate: 85.1% [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 198.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.0%, Prefix cache hit rate: 85.1% [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:42 [loggers.py:259] Engine 000: Avg prompt throughput: 209.6 tokens/s, Avg generation throughput: 201.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 84.9% [repeated 47x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 8x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 258.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.5%, Prefix cache hit rate: 84.9% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 17x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:53 [loggers.py:259] Engine 000: Avg prompt throughput: 23.6 tokens/s, Avg generation throughput: 225.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 85.4% [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 11x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:51:58 [loggers.py:259] Engine 000: Avg prompt throughput: 35.0 tokens/s, Avg generation throughput: 236.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 85.6% [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 25x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:52:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 191.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.3%, Prefix cache hit rate: 85.6% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:52:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 85.6% [repeated 47x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 30x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:52:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 155.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.1%, Prefix cache hit rate: 85.6% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 19x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:52:18 [loggers.py:259] Engine 000: Avg prompt throughput: 235.2 tokens/s, Avg generation throughput: 169.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 84.9% [repeated 48x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 25x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:52:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 141.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 84.9% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 23x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52) INFO 06-08 20:52:28 [loggers.py:259] Engine 000: Avg prompt throughput: 165.8 tokens/s, Avg generation throughput: 164.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 85.3% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) INFO 06-08 20:52:33 [loggers.py:259] Engine 000: Avg prompt throughput: 27.5 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 86.3% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) INFO 06-08 20:52:38 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 91.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 86.3% [repeated 47x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58) INFO 06-08 20:52:43 [loggers.py:259] Engine 000: Avg prompt throughput: 20.9 tokens/s, Avg generation throughput: 91.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 86.6% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 28x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59) INFO 06-08 20:52:48 [loggers.py:259] Engine 000: Avg prompt throughput: 13.9 tokens/s, Avg generation throughput: 213.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 93.7% [repeated 49x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59) INFO 06-08 20:52:53 [loggers.py:259] Engine 000: Avg prompt throughput: 9.6 tokens/s, Avg generation throughput: 163.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.4%, Prefix cache hit rate: 93.9% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) [fd-monitor] [20:52:54] OK: 266 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:52:54] OK: RSS 1.15 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 19x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59) INFO 06-08 20:52:58 [loggers.py:259] Engine 000: Avg prompt throughput: 9.6 tokens/s, Avg generation throughput: 148.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 94.1% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 29x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:53:01] OK: 408 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:53:01] OK: RSS 1.17 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 450.0 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59) INFO 06-08 20:53:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 179.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 94.1% [repeated 48x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) INFO 06-08 20:53:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 253.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 87.7% [repeated 49x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:53:10] OK: 379 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:53:10] OK: RSS 1.19 GiB | node mem 386.4/858.0 GiB used (45.0%), avail 471.6 GiB (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) INFO 06-08 20:53:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 87.7% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 8x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) INFO 06-08 20:53:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 87.7% [repeated 48x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:53:18] OK: 275 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:53:18] OK: RSS 1.15 GiB | node mem 423.9/858.0 GiB used (49.4%), avail 434.1 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61) INFO 06-08 20:53:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 87.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:28 [loggers.py:259] Engine 000: Avg prompt throughput: 60.3 tokens/s, Avg generation throughput: 165.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 82.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (skyrl_entrypoint pid=546644) [fd-monitor] [20:53:32] OK: 339 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:53:32] OK: RSS 1.79 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:33 [loggers.py:259] Engine 000: Avg prompt throughput: 63.3 tokens/s, Avg generation throughput: 191.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.0%, Prefix cache hit rate: 82.6% [repeated 48x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 36x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:38 [loggers.py:259] Engine 000: Avg prompt throughput: 56.1 tokens/s, Avg generation throughput: 196.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 83.8% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:43 [loggers.py:259] Engine 000: Avg prompt throughput: 187.6 tokens/s, Avg generation throughput: 233.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 41x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 205.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 20x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 170.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 84.5% [repeated 49x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 17x across cluster] (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:53:58 [loggers.py:259] Engine 000: Avg prompt throughput: 64.9 tokens/s, Avg generation throughput: 191.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 213.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:08 [loggers.py:259] Engine 000: Avg prompt throughput: 52.8 tokens/s, Avg generation throughput: 254.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 31x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 217.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 20x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:18 [loggers.py:259] Engine 000: Avg prompt throughput: 106.0 tokens/s, Avg generation throughput: 193.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 15x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 213.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 13x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 213.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:54:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 284.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 85.8% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:54:38 [loggers.py:259] Engine 000: Avg prompt throughput: 33.5 tokens/s, Avg generation throughput: 317.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 85.8% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new (RolloutCoordinator pid=2690439, ip=10.128.32.54) LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:54:43 [loggers.py:259] Engine 000: Avg prompt throughput: 68.9 tokens/s, Avg generation throughput: 367.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 85.7% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 25x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new (RolloutCoordinator pid=2690439, ip=10.128.32.54) LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:54:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 378.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 85.7% [repeated 48x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 31x across cluster] (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:54:53 [loggers.py:259] Engine 000: Avg prompt throughput: 60.8 tokens/s, Avg generation throughput: 398.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 85.5% [repeated 48x across cluster] (RolloutCoordinator pid=547199) [fd-monitor] [20:54:54] OK: 299 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=547199) [fd-monitor] [20:54:54] OK: RSS 1.20 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.2 GiB (RolloutCoordinator pid=3120643, ip=10.128.32.50) Provider List: https://docs.litellm.ai/docs/providers [repeated 2x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:54:58 [loggers.py:259] Engine 000: Avg prompt throughput: 73.8 tokens/s, Avg generation throughput: 305.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 8x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:55:01] OK: 431 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=321702, ip=10.128.32.56) [fd-monitor] [20:55:01] OK: RSS 1.21 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 449.9 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:55:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 297.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 7x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:55:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 298.0 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:55:10] OK: 407 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=2690439, ip=10.128.32.54) [fd-monitor] [20:55:10] OK: RSS 1.20 GiB | node mem 386.4/858.0 GiB used (45.0%), avail 471.6 GiB (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new (RolloutCoordinator pid=2690439, ip=10.128.32.54) LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 28x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (RolloutCoordinator pid=3120643, ip=10.128.32.50) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:55:13 [loggers.py:259] Engine 000: Avg prompt throughput: 45.8 tokens/s, Avg generation throughput: 305.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.6%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 12x across cluster] (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=547199) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:55:18] OK: 414 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=3120643, ip=10.128.32.50) [fd-monitor] [20:55:18] OK: RSS 1.20 GiB | node mem 423.8/858.0 GiB used (49.4%), avail 434.1 GiB (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:55:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 310.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 10x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new (RolloutCoordinator pid=2690439, ip=10.128.32.54) LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55) INFO 06-08 20:55:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 299.4 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 84.5% [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:55:28 [loggers.py:259] Engine 000: Avg prompt throughput: 198.0 tokens/s, Avg generation throughput: 404.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.2%, Prefix cache hit rate: 85.2% [repeated 49x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=321702, ip=10.128.32.56) (skyrl_entrypoint pid=546644) [fd-monitor] [20:55:32] OK: 458 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=546644) [fd-monitor] [20:55:32] OK: RSS 1.80 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.2 GiB (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:55:33 [loggers.py:259] Engine 000: Avg prompt throughput: 240.9 tokens/s, Avg generation throughput: 419.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.6%, Prefix cache hit rate: 84.8% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:55:38 [loggers.py:259] Engine 000: Avg prompt throughput: 236.9 tokens/s, Avg generation throughput: 396.2 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 84.9% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 5x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:55:43 [loggers.py:259] Engine 000: Avg prompt throughput: 87.2 tokens/s, Avg generation throughput: 425.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.6%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new (RolloutCoordinator pid=2690439, ip=10.128.32.54) LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'. (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) Provider List: https://docs.litellm.ai/docs/providers [repeated 11x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=2690439, ip=10.128.32.54) (RolloutCoordinator pid=321702, ip=10.128.32.56) (RolloutCoordinator pid=321702, ip=10.128.32.56) (AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53) INFO 06-08 20:55:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 378.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 84.7% [repeated 48x across cluster] (RolloutCoordinator pid=321702, ip=10.128.32.56) Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new (RolloutCoordinator pid=321702, ip=10.128.32.56) LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'. (RolloutCoordinator pid=2690439, ip=10.128.32.54) Provider List: https://docs.litellm.ai/docs/providers [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61) INFO 06-08 20:55:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 87.7% [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51) INFO 06-08 20:55:54 [loggers.py:259] Engine 000: Avg prompt throughput: 552.6 tokens/s, Avg generation throughput: 360.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 85.9% [repeated 4x across cluster] Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqnorm', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqnorm', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true'] Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in main() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() ^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( ^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run _ = ret.return_value ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) ^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main ray.get(skyrl_entrypoint.remote(cfg)) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get values, debugger_breakpoint = worker.get_objects( ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1017, in get_objects raise value ray.exceptions.NodeDiedError: Task failed because the node it was running on is dead or unavailable. Node IP: 10.128.32.49, node ID: 4cba01b979f7f003d04abf087b90b4e01890df9fe091283ac00c4962. This can happen if the node was preempted, had a hardware failure, or its raylet crashed unexpectedly. To see node death information, use `ray list nodes --filter node_id=4cba01b979f7f003d04abf087b90b4e01890df9fe091283ac00c4962`, check the Ray dashboard cluster page, search the node ID in the GCS logs, or use `ray logs raylet.out -ip 10.128.32.49`. Collecting Ray logs from worker jpbo-042-02... WARNING: Failed to collect logs from jpbo-042-02 Collecting Ray logs from worker jpbo-042-03... Stopping Ray cluster... WARNING: Failed to collect logs from jpbo-042-03 Collecting Ray logs from worker jpbo-042-04... WARNING: Failed to collect logs from jpbo-042-04 Collecting Ray logs from worker jpbo-042-05... WARNING: Failed to collect logs from jpbo-042-05 Collecting Ray logs from worker jpbo-042-06... Ray cluster stopped [RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)... WARNING: Failed to collect logs from jpbo-042-06 Collecting Ray logs from worker jpbo-042-07... Collecting Ray logs from worker jpbo-042-02... WARNING: Failed to collect logs from jpbo-042-07 Collecting Ray logs from worker jpbo-042-08... WARNING: Failed to collect logs from jpbo-042-02 Collecting Ray logs from worker jpbo-042-03... WARNING: Failed to collect logs from jpbo-042-03 Collecting Ray logs from worker jpbo-042-04... WARNING: Failed to collect logs from jpbo-042-04 Collecting Ray logs from worker jpbo-042-05... WARNING: Failed to collect logs from jpbo-042-08 Collecting Ray logs from worker jpbo-042-09... WARNING: Failed to collect logs from jpbo-042-05 Collecting Ray logs from worker jpbo-042-06... WARNING: Failed to collect logs from jpbo-042-06 Collecting Ray logs from worker jpbo-042-07... WARNING: Failed to collect logs from jpbo-042-07 Collecting Ray logs from worker jpbo-042-08... WARNING: Failed to collect logs from jpbo-042-08 Collecting Ray logs from worker jpbo-042-09... WARNING: Failed to collect logs from jpbo-042-09 Collecting Ray logs from worker jpbo-042-10... WARNING: Failed to collect logs from jpbo-042-09 Collecting Ray logs from worker jpbo-042-10... WARNING: Failed to collect logs from jpbo-042-10 WARNING: Failed to collect logs from jpbo-042-10 Collecting Ray logs from worker jpbo-042-11... Collecting Ray logs from worker jpbo-042-11... WARNING: Failed to collect logs from jpbo-042-11 Collecting Ray logs from worker jpbo-042-12... WARNING: Failed to collect logs from jpbo-042-11 Collecting Ray logs from worker jpbo-042-12... WARNING: Failed to collect logs from jpbo-042-12 Collecting Ray logs from worker jpbo-042-13... WARNING: Failed to collect logs from jpbo-042-12 Collecting Ray logs from worker jpbo-042-13... WARNING: Failed to collect logs from jpbo-042-13 Collecting Ray logs from worker jpbo-042-14... WARNING: Failed to collect logs from jpbo-042-13 Collecting Ray logs from worker jpbo-042-14... WARNING: Failed to collect logs from jpbo-042-14 Ray log preservation complete WARNING: Failed to collect logs from jpbo-042-14 [RLJobRunner] Crash-time Ray log preservation complete. [RLJobRunner] Launching trace upload (training exit code: 1): repo_id: DCAgent/ablation-pymethods2test-seqnorm job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm episodes: last log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/logs/ablation-pymethods2test-seqnorm_trace_upload.log [RLJobRunner] Waiting for trace upload to complete...