8686 lines
676 KiB
Plaintext
8686 lines
676 KiB
Plaintext
|
||
The following have been reloaded with a version change:
|
||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||
|
||
|
||
Lmod is automatically replacing "GCC/14.3.0" with
|
||
"nvidia-compilers/25.9-CUDA-13".
|
||
|
||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
[ray] RAY_TMPDIR=/tmp/ray/ray_639476
|
||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_639476
|
||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_639476
|
||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||
[proxy] Tunnel port: 7003
|
||
[proxy] Node IP: 10.128.32.49 (workers will connect here)
|
||
[proxy] ✓ SSH tunnel started successfully
|
||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_639476.conf
|
||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||
[proxy] ✓ Daytona timeout settings configured
|
||
[proxy] Testing proxy connectivity...
|
||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_639476.conf
|
||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||
[proxy] ⚠ Tunnel not accessible at 10.128.32.49:7003 (workers may fail)
|
||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||
=== Universal RL Training Runner ===
|
||
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/configs/ablation-pymethods2test-seqnorm_rl_config.json
|
||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Python version: Python 3.12.12
|
||
UV_USE_IO_URING: 0
|
||
Proxy: DISABLED (direct internet or not configured)
|
||
========================================
|
||
=== RLJobRunner: ablation-pymethods2test-seqnorm ===
|
||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/wandb
|
||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/wandb
|
||
HF_TOKEN=****pDbg
|
||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||
Environment configured:
|
||
TENSOR_PARALLEL_SIZE=1
|
||
NUM_INFERENCE_ENGINES=56
|
||
POLICY_NUM_NODES=14
|
||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/wandb
|
||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||
Cleaning up existing Ray instances...
|
||
=== Starting Ray Cluster ===
|
||
Nodes: 14
|
||
GPUs per node: 4
|
||
CPUs per node: 288
|
||
Head node: jpbo-042-01 (10.128.32.49)
|
||
Ray port: 6379
|
||
============================
|
||
Starting Ray head on jpbo-042-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-042-01.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.49 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.49 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray head on jpbo-042-01
|
||
Starting Ray worker on jpbo-042-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-02.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 1 on jpbo-042-02
|
||
Starting Ray worker on jpbo-042-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-03.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 2 on jpbo-042-03
|
||
Starting Ray worker on jpbo-042-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-04.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 3 on jpbo-042-04
|
||
Starting Ray worker on jpbo-042-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-05.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 4 on jpbo-042-05
|
||
Starting Ray worker on jpbo-042-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-06.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.54 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.54 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 5 on jpbo-042-06
|
||
Starting Ray worker on jpbo-042-07 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-07.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.55 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.55 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 6 on jpbo-042-07
|
||
Starting Ray worker on jpbo-042-08 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-08.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.56 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.56 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 7 on jpbo-042-08
|
||
Starting Ray worker on jpbo-042-09 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-09.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.57 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-09 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.57 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 8 on jpbo-042-09
|
||
Starting Ray worker on jpbo-042-10 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-10.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.58 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-10 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.58 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 9 on jpbo-042-10
|
||
Starting Ray worker on jpbo-042-11 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-11.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.59 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.59 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 10 on jpbo-042-11
|
||
Starting Ray worker on jpbo-042-12 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-12.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.60 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.60 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 11 on jpbo-042-12
|
||
Starting Ray worker on jpbo-042-13 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-13.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.61 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.61 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 12 on jpbo-042-13
|
||
Starting Ray worker on jpbo-042-14 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-14.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.62 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-14 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.49:6379 --node-ip-address=10.128.32.62 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 13 on jpbo-042-14
|
||
Waiting for cluster (56 GPUs, 14 nodes)...
|
||
Connecting to Ray at 10.128.32.49:6379 (expecting 14 nodes, 56.0 GPUs)
|
||
Ray connection established, polling for resources...
|
||
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'object_store_memory': 601295421440.0, 'CPU': 4032.0, 'accelerator_type:GH200': 14.0, 'node:10.128.32.50': 1.0, 'memory': 10479941910528.0, 'GPU': 56.0, 'node:10.128.32.58': 1.0, 'node:10.128.32.60': 1.0, 'node:10.128.32.62': 1.0, 'node:10.128.32.57': 1.0, 'node:10.128.32.55': 1.0, 'node:10.128.32.51': 1.0, 'node:10.128.32.52': 1.0, 'node:10.128.32.59': 1.0, 'node:10.128.32.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.54': 1.0, 'node:10.128.32.61': 1.0, 'node:10.128.32.53': 1.0, 'node:10.128.32.56': 1.0}
|
||
✓ Ray cluster ready
|
||
=== Ray Cluster Ready ===
|
||
Address: 10.128.32.49:6379
|
||
Total GPUs: 56
|
||
=========================
|
||
Ray cluster ready at 10.128.32.49:6379
|
||
Total GPUs available: 56
|
||
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
|
||
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
|
||
|
||
Running SkyRL:
|
||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
|
||
Args: 120 Hydra arguments
|
||
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
|
||
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
|
||
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqnorm ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqnorm trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
|
||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_639476.conf
|
||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
2026-06-08 20:41:23.641 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:765 - Exporting wandb api key to ray runtime env
|
||
2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:784 - Exporting RAY_ADDRESS to ray runtime env
|
||
2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:809 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
|
||
2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:809 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
|
||
2026-06-08 20:41:23.642 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:809 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
|
||
2026-06-08 20:41:23,642 INFO worker.py:1680 -- Using address 10.128.32.49:6379 set in the environment variable RAY_ADDRESS
|
||
2026-06-08 20:41:23,675 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.49:6379...
|
||
2026-06-08 20:41:23,686 INFO worker.py:2007 -- Connected to Ray cluster.
|
||
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||
warnings.warn(
|
||
[33m(raylet, ip=10.128.32.54)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
2026-06-08 20:41:25.873 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
|
||
[36m(pid=530083)[0m [2026-06-08 20:41:27,275 E 530083 530758] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:41:32.678[0m | [1mINFO [0m | [36mskyrl_train.entrypoints.main_base[0m:[36m_configure_log_level[0m:[36m214[0m - [1mSkyRL log level set to: INFO[0m
|
||
[36m(pid=530096)[0m [2026-06-08 20:41:29,941 E 530096 531589] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 17x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:41:32.962[0m | [1mINFO [0m | [36mexamples.terminal_bench.dataset[0m:[36m_load_data_files[0m:[36m40[0m - [1mLoading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:41:34.762[0m | [1mINFO [0m | [36mexamples.terminal_bench.dataset[0m:[36m_load_data_files[0m:[36m50[0m - [1mFound 5000 valid task directories out of 5000 total directories[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:41:34.763[0m | [1mINFO [0m | [36mexamples.terminal_bench.dataset[0m:[36m__init__[0m:[36m27[0m - [1mTerminalBenchTaskDataset initialized with 5000 task paths[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:41:34.775[0m | [1mINFO [0m | [36mskyrl_train.entrypoints.main_base[0m:[36m_setup_trainer[0m:[36m407[0m - [1mdata:
|
||
[36m(skyrl_entrypoint pid=546644)[0m train_data:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
|
||
[36m(skyrl_entrypoint pid=546644)[0m val_data: []
|
||
[36m(skyrl_entrypoint pid=546644)[0m trainer:
|
||
[36m(skyrl_entrypoint pid=546644)[0m placement:
|
||
[36m(skyrl_entrypoint pid=546644)[0m colocate_all: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m colocate_policy_ref: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_num_nodes: 2
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_num_gpus_per_node: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m critic_num_nodes: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m critic_num_gpus_per_node: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m ref_num_nodes: 2
|
||
[36m(skyrl_entrypoint pid=546644)[0m ref_num_gpus_per_node: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_strict_spread_pg: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_per_gpu_bundles: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_force_cvd_mask: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m sequence_parallel_backend: ulysses
|
||
[36m(skyrl_entrypoint pid=546644)[0m strategy: fsdp2
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy:
|
||
[36m(skyrl_entrypoint pid=546644)[0m model:
|
||
[36m(skyrl_entrypoint pid=546644)[0m path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
|
||
[36m(skyrl_entrypoint pid=546644)[0m lora:
|
||
[36m(skyrl_entrypoint pid=546644)[0m rank: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m alpha: 16
|
||
[36m(skyrl_entrypoint pid=546644)[0m dropout: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m lora_sync_path: /tmp/skyrl_lora_sync
|
||
[36m(skyrl_entrypoint pid=546644)[0m target_modules: all-linear
|
||
[36m(skyrl_entrypoint pid=546644)[0m exclude_modules: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepspeed_config: ${deepspeed_config.train}
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer: AdamW
|
||
[36m(skyrl_entrypoint pid=546644)[0m lr: 8.0e-06
|
||
[36m(skyrl_entrypoint pid=546644)[0m adam_betas:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - 0.9
|
||
[36m(skyrl_entrypoint pid=546644)[0m - 0.999
|
||
[36m(skyrl_entrypoint pid=546644)[0m weight_decay: 0.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_grad_norm: 0.9
|
||
[36m(skyrl_entrypoint pid=546644)[0m offload_after_step: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m num_warmup_steps: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m scheduler: constant_with_warmup
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m fsdp_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m reshard_after_forward: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m fsdp_size: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_token_dispatcher_type: alltoall
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_router_replay: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_grouped_gemm: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m ep_comm_backend: torch
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepep_num_sms: 20
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepep_token_chunk_size: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m sequence_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_torch_compile: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m record_memory: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m megatron_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m tensor_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m pipeline_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m context_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_tensor_parallel_size: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m ddp_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m grad_reduce_in_fp32: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m overlap_grad_reduce: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m overlap_param_gather: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m average_in_collective: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m model_config_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m torch_profiler_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m ranks: []
|
||
[36m(skyrl_entrypoint pid=546644)[0m save_path: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_config_kwargs:
|
||
[36m(skyrl_entrypoint pid=546644)[0m overlap_cpu_optimizer_d2h_h2d: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_precision_aware_optimizer: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_offload_fraction: 0.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m transformer_config_kwargs:
|
||
[36m(skyrl_entrypoint pid=546644)[0m recompute_granularity: full
|
||
[36m(skyrl_entrypoint pid=546644)[0m recompute_modules:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - core_attn
|
||
[36m(skyrl_entrypoint pid=546644)[0m recompute_method: uniform
|
||
[36m(skyrl_entrypoint pid=546644)[0m recompute_num_layers: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m empty_cuda_cache: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m ref:
|
||
[36m(skyrl_entrypoint pid=546644)[0m model:
|
||
[36m(skyrl_entrypoint pid=546644)[0m path: ${trainer.policy.model.path}
|
||
[36m(skyrl_entrypoint pid=546644)[0m sequence_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepspeed_config: ${deepspeed_config.eval}
|
||
[36m(skyrl_entrypoint pid=546644)[0m fsdp_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m reshard_after_forward: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m fsdp_size: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_token_dispatcher_type: alltoall
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_router_replay: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_grouped_gemm: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m ep_comm_backend: torch
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepep_num_sms: 20
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepep_token_chunk_size: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m megatron_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m tensor_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m pipeline_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m context_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m model_config_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m transformer_config_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m critic:
|
||
[36m(skyrl_entrypoint pid=546644)[0m model:
|
||
[36m(skyrl_entrypoint pid=546644)[0m path: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m lora:
|
||
[36m(skyrl_entrypoint pid=546644)[0m rank: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m alpha: 16
|
||
[36m(skyrl_entrypoint pid=546644)[0m dropout: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m target_modules: all-linear
|
||
[36m(skyrl_entrypoint pid=546644)[0m exclude_modules: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepspeed_config: ${deepspeed_config.train}
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer: AdamW
|
||
[36m(skyrl_entrypoint pid=546644)[0m lr: 5.0e-06
|
||
[36m(skyrl_entrypoint pid=546644)[0m adam_betas:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - 0.9
|
||
[36m(skyrl_entrypoint pid=546644)[0m - 0.999
|
||
[36m(skyrl_entrypoint pid=546644)[0m weight_decay: 0.01
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_grad_norm: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m offload_after_step: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m num_warmup_steps: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m scheduler: constant_with_warmup
|
||
[36m(skyrl_entrypoint pid=546644)[0m optimizer_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m fsdp_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m reshard_after_forward: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m fsdp_size: -1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_token_dispatcher_type: alltoall
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_router_replay: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m moe_grouped_gemm: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m ep_comm_backend: torch
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepep_num_sms: 20
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepep_token_chunk_size: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m sequence_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m algorithm:
|
||
[36m(skyrl_entrypoint pid=546644)[0m advantage_estimator: rloo_n
|
||
[36m(skyrl_entrypoint pid=546644)[0m kl_ctrl:
|
||
[36m(skyrl_entrypoint pid=546644)[0m type: fixed
|
||
[36m(skyrl_entrypoint pid=546644)[0m kl_target: 0.1
|
||
[36m(skyrl_entrypoint pid=546644)[0m horizon: 10000
|
||
[36m(skyrl_entrypoint pid=546644)[0m kl_estimator_type: k3
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_kl_estimator_k3: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_abs_kl: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_kl_in_reward: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_kl_loss: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m kl_loss_coef: 0.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_entropy_loss: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m entropy_loss_coef: 0.01
|
||
[36m(skyrl_entrypoint pid=546644)[0m advantage_batch_normalize: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m value_head_prefix: value_head
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_loss_type: regular
|
||
[36m(skyrl_entrypoint pid=546644)[0m loss_reduction: seq_mean_token_sum_norm_global
|
||
[36m(skyrl_entrypoint pid=546644)[0m global_loss_denom: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m grpo_norm_by_std: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m rloo_n_min_group_size: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m rloo_n_filter_zero_reward_groups: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m lambd: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m gamma: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m eps_clip_low: 0.2
|
||
[36m(skyrl_entrypoint pid=546644)[0m eps_clip_high: 0.05
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_ratio_c: 3.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m tis_imp_ratio_cap: -1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_tis: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m sapo:
|
||
[36m(skyrl_entrypoint pid=546644)[0m tau_pos: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m tau_neg: 1.05
|
||
[36m(skyrl_entrypoint pid=546644)[0m value_clip: 0.2
|
||
[36m(skyrl_entrypoint pid=546644)[0m dynamic_sampling:
|
||
[36m(skyrl_entrypoint pid=546644)[0m type: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_sample_batches: 30
|
||
[36m(skyrl_entrypoint pid=546644)[0m min_replace_ratio: 0.3
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_cov:
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_ratio: 0.0002
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_cov_lb: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_cov_ub: 5.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m kl_cov:
|
||
[36m(skyrl_entrypoint pid=546644)[0m kl_cov_frac: 0.2
|
||
[36m(skyrl_entrypoint pid=546644)[0m ppo_kl_coef: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m cispo:
|
||
[36m(skyrl_entrypoint pid=546644)[0m cispo_eps_clip_low: 0
|
||
[36m(skyrl_entrypoint pid=546644)[0m cispo_eps_clip_high: 5
|
||
[36m(skyrl_entrypoint pid=546644)[0m z_clip:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m alpha: 0.97
|
||
[36m(skyrl_entrypoint pid=546644)[0m z_thresh: 2.5
|
||
[36m(skyrl_entrypoint pid=546644)[0m warmup_steps: 25
|
||
[36m(skyrl_entrypoint pid=546644)[0m mode: zscore
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_option: adaptive_scaling
|
||
[36m(skyrl_entrypoint pid=546644)[0m clip_factor: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m skip_update_on_spike: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m stale_clip:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m alpha: 0.3
|
||
[36m(skyrl_entrypoint pid=546644)[0m entropy_threshold: 0.15
|
||
[36m(skyrl_entrypoint pid=546644)[0m entropy_window: 10
|
||
[36m(skyrl_entrypoint pid=546644)[0m min_lr_scale: 0.1
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_seq_len: 1004095
|
||
[36m(skyrl_entrypoint pid=546644)[0m fully_async:
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_staleness_steps: 16
|
||
[36m(skyrl_entrypoint pid=546644)[0m num_parallel_generation_workers: 338
|
||
[36m(skyrl_entrypoint pid=546644)[0m gradient_checkpointing: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m gradient_checkpointing_use_reentrant: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m seed: 42
|
||
[36m(skyrl_entrypoint pid=546644)[0m resume_mode: latest
|
||
[36m(skyrl_entrypoint pid=546644)[0m resume_path: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_ckpts_to_keep: -1
|
||
[36m(skyrl_entrypoint pid=546644)[0m ckpt_interval: 2
|
||
[36m(skyrl_entrypoint pid=546644)[0m hf_save_interval: 5
|
||
[36m(skyrl_entrypoint pid=546644)[0m hf_upload_mode: latest
|
||
[36m(skyrl_entrypoint pid=546644)[0m export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports
|
||
[36m(skyrl_entrypoint pid=546644)[0m bf16: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m epochs: 2
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_steps: 80
|
||
[36m(skyrl_entrypoint pid=546644)[0m update_epochs_per_batch: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m train_batch_size: 64
|
||
[36m(skyrl_entrypoint pid=546644)[0m policy_mini_batch_size: 64
|
||
[36m(skyrl_entrypoint pid=546644)[0m critic_mini_batch_size: 256
|
||
[36m(skyrl_entrypoint pid=546644)[0m micro_train_batch_size_per_gpu: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m micro_forward_batch_size_per_gpu: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m update_ref_every_epoch: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_sample_packing: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m eval_batch_size: 64
|
||
[36m(skyrl_entrypoint pid=546644)[0m eval_before_train: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m eval_interval: 999999
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_prompt_length: 999999
|
||
[36m(skyrl_entrypoint pid=546644)[0m flash_attn: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m disable_fast_tokenizer: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m target_modules: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m exclude_modules: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m project_name: OpenThoughts-Agent
|
||
[36m(skyrl_entrypoint pid=546644)[0m run_name: ablation-pymethods2test-seqnorm
|
||
[36m(skyrl_entrypoint pid=546644)[0m logger: console
|
||
[36m(skyrl_entrypoint pid=546644)[0m tracker_commit_each_step: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m dump_data_batch: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m dump_eval_results: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m log_level: INFO
|
||
[36m(skyrl_entrypoint pid=546644)[0m rope_scaling: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m rope_theta: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m step_wise_training: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m hf_hub_repo_id: laion/ablation-pymethods2test-seqnorm
|
||
[36m(skyrl_entrypoint pid=546644)[0m hf_hub_private: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m hf_hub_revision: main
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_db_registration: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m generator:
|
||
[36m(skyrl_entrypoint pid=546644)[0m model_name: ${trainer.policy.model.path}
|
||
[36m(skyrl_entrypoint pid=546644)[0m model_dtype: bfloat16
|
||
[36m(skyrl_entrypoint pid=546644)[0m timeout_multiplier: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m run_engines_locally: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m num_inference_engines: 48
|
||
[36m(skyrl_entrypoint pid=546644)[0m backend: vllm
|
||
[36m(skyrl_entrypoint pid=546644)[0m weight_sync_backend: nccl
|
||
[36m(skyrl_entrypoint pid=546644)[0m fuse_weights: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m weight_transfer_threshold_cuda_ipc_GB: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_pipeline_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_expert_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_data_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_mp_backend: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m n_samples_per_prompt: 8
|
||
[36m(skyrl_entrypoint pid=546644)[0m async_engine: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m batched: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_input_length: ${trainer.max_prompt_length}
|
||
[36m(skyrl_entrypoint pid=546644)[0m vllm_v1_disable_multiproc: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_prefix_caching: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_chunked_prefill: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_num_batched_tokens: 65536
|
||
[36m(skyrl_entrypoint pid=546644)[0m enforce_eager: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m fully_sharded_loras: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_ray_prometheus_stats: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m vllm_stats_interval: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m gpu_memory_utilization: 0.75
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_num_seqs: 24
|
||
[36m(skyrl_entrypoint pid=546644)[0m remote_inference_engine_urls:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - 127.0.0.1:8001
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_http_endpoint: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m http_endpoint_host: 127.0.0.1
|
||
[36m(skyrl_entrypoint pid=546644)[0m http_endpoint_port: 8000
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_turns: 999999
|
||
[36m(skyrl_entrypoint pid=546644)[0m chat_template:
|
||
[36m(skyrl_entrypoint pid=546644)[0m source: name
|
||
[36m(skyrl_entrypoint pid=546644)[0m name_or_path: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m chat_template_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m engine_init_kwargs:
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_model_len: 32768
|
||
[36m(skyrl_entrypoint pid=546644)[0m custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(skyrl_entrypoint pid=546644)[0m served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
|
||
[36m(skyrl_entrypoint pid=546644)[0m override_existing_update_group: disable
|
||
[36m(skyrl_entrypoint pid=546644)[0m sampling_params:
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_generate_length: 4096
|
||
[36m(skyrl_entrypoint pid=546644)[0m repetition_penalty: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m temperature: 0.7
|
||
[36m(skyrl_entrypoint pid=546644)[0m top_p: 0.95
|
||
[36m(skyrl_entrypoint pid=546644)[0m min_p: 0.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m top_k: 20
|
||
[36m(skyrl_entrypoint pid=546644)[0m logprobs: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m stop: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m use_conversation_multi_turn: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m append_eos_token_after_stop_str_in_multi_turn: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m eval_sampling_params:
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_generate_length: ${generator.sampling_params.max_generate_length}
|
||
[36m(skyrl_entrypoint pid=546644)[0m repetition_penalty: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m temperature: 0.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m top_p: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m min_p: 0.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m top_k: -1
|
||
[36m(skyrl_entrypoint pid=546644)[0m logprobs: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m stop: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m eval_n_samples_per_prompt: 8
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_reward_on_non_stop: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m apply_overlong_filtering: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m rope_scaling: ${trainer.rope_scaling}
|
||
[36m(skyrl_entrypoint pid=546644)[0m rope_theta: ${trainer.rope_theta}
|
||
[36m(skyrl_entrypoint pid=546644)[0m teacher:
|
||
[36m(skyrl_entrypoint pid=546644)[0m model_path: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m top_k_logprobs: 256
|
||
[36m(skyrl_entrypoint pid=546644)[0m num_inference_engines: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m inference_engine_pipeline_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m gpu_memory_utilization: 0.9
|
||
[36m(skyrl_entrypoint pid=546644)[0m enforce_eager: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m backend: vllm
|
||
[36m(skyrl_entrypoint pid=546644)[0m engine_init_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=546644)[0m environment:
|
||
[36m(skyrl_entrypoint pid=546644)[0m env_class: gsm8k
|
||
[36m(skyrl_entrypoint pid=546644)[0m skyrl_gym:
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_env_workers: 32
|
||
[36m(skyrl_entrypoint pid=546644)[0m text2sql:
|
||
[36m(skyrl_entrypoint pid=546644)[0m db_path: /home/ray/default/sql_data
|
||
[36m(skyrl_entrypoint pid=546644)[0m llm_as_a_judge:
|
||
[36m(skyrl_entrypoint pid=546644)[0m model: gpt-4o-mini
|
||
[36m(skyrl_entrypoint pid=546644)[0m base_url: null
|
||
[36m(skyrl_entrypoint pid=546644)[0m search:
|
||
[36m(skyrl_entrypoint pid=546644)[0m log_requests: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m search_url: http://127.0.0.1:8000/retrieve
|
||
[36m(skyrl_entrypoint pid=546644)[0m topk: 3
|
||
[36m(skyrl_entrypoint pid=546644)[0m timeout: 30
|
||
[36m(skyrl_entrypoint pid=546644)[0m rollout:
|
||
[36m(skyrl_entrypoint pid=546644)[0m fanout:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m num_coordinators: 4
|
||
[36m(skyrl_entrypoint pid=546644)[0m cpus_per_coordinator: 8
|
||
[36m(skyrl_entrypoint pid=546644)[0m deepspeed_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m train:
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_optimization:
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage: 3
|
||
[36m(skyrl_entrypoint pid=546644)[0m offload_param:
|
||
[36m(skyrl_entrypoint pid=546644)[0m device: none
|
||
[36m(skyrl_entrypoint pid=546644)[0m offload_optimizer:
|
||
[36m(skyrl_entrypoint pid=546644)[0m device: none
|
||
[36m(skyrl_entrypoint pid=546644)[0m pin_memory: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m sub_group_size: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m reduce_bucket_size: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage3_param_persistence_threshold: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage3_prefetch_bucket_size: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage3_max_live_parameters: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage3_max_reuse_distance: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m round_robin_gradients: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_hpz_partition_size: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_quantized_weights: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_quantized_gradients: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m torch_autocast:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m dtype: bfloat16
|
||
[36m(skyrl_entrypoint pid=546644)[0m disable_trace_cache: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m data_types:
|
||
[36m(skyrl_entrypoint pid=546644)[0m grad_accum_dtype: fp32
|
||
[36m(skyrl_entrypoint pid=546644)[0m gradient_clipping: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m wall_clock_breakdown: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m prescale_gradient: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m eval:
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_optimization:
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage: 3
|
||
[36m(skyrl_entrypoint pid=546644)[0m stage3_param_persistence_threshold: auto
|
||
[36m(skyrl_entrypoint pid=546644)[0m offload_param:
|
||
[36m(skyrl_entrypoint pid=546644)[0m device: cpu
|
||
[36m(skyrl_entrypoint pid=546644)[0m pin_memory: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m torch_autocast:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m dtype: bfloat16
|
||
[36m(skyrl_entrypoint pid=546644)[0m gradient_clipping: 1.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m prescale_gradient: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m wall_clock_breakdown: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m terminal_bench_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/trace_jobs
|
||
[36m(skyrl_entrypoint pid=546644)[0m harbor:
|
||
[36m(skyrl_entrypoint pid=546644)[0m name: terminus-2
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_episodes: 999999
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_summarize: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m store_all_messages: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m trajectory_config:
|
||
[36m(skyrl_entrypoint pid=546644)[0m raw_content: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_episode_logging: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m record_terminal_session: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_pane_logging: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m strict_json_parser: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m interleaved_thinking: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m extra_body:
|
||
[36m(skyrl_entrypoint pid=546644)[0m chat_template_kwargs:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_thinking: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m override_timeout_sec: 900
|
||
[36m(skyrl_entrypoint pid=546644)[0m override_cpus: 1
|
||
[36m(skyrl_entrypoint pid=546644)[0m override_memory_mb: 2048
|
||
[36m(skyrl_entrypoint pid=546644)[0m override_storage_mb: 2048
|
||
[36m(skyrl_entrypoint pid=546644)[0m auto_snapshot: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m verifier_override_timeout_sec: 120
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_retries: 3
|
||
[36m(skyrl_entrypoint pid=546644)[0m min_wait_sec: 60.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_wait_sec: 600.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m wait_multiplier: 2.0
|
||
[36m(skyrl_entrypoint pid=546644)[0m exclude_exceptions:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - VerifierTimeoutError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - VerifierRuntimeError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - RewardFileNotFoundError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - RewardFileEmptyError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - VerifierOutputParseError
|
||
[36m(skyrl_entrypoint pid=546644)[0m n_concurrent_trials: 675
|
||
[36m(skyrl_entrypoint pid=546644)[0m log_level: INFO
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_reward_shaping: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m enable_error_classification: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m mask_exceptions:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - DaytonaError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - EnvironmentStartTimeoutError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - NetworkError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - ConnectionError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - RewardFileNotFoundError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - RewardFileEmptyError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - AgentEnvironmentTimeoutError
|
||
[36m(skyrl_entrypoint pid=546644)[0m - ContextLengthExceededError
|
||
[36m(skyrl_entrypoint pid=546644)[0m default_error_treatment: zero
|
||
[36m(skyrl_entrypoint pid=546644)[0m passthrough_exceptions:
|
||
[36m(skyrl_entrypoint pid=546644)[0m - AgentTimeoutError
|
||
[36m(skyrl_entrypoint pid=546644)[0m zero_exceptions: []
|
||
[36m(skyrl_entrypoint pid=546644)[0m model_info:
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_input_tokens: 32000
|
||
[36m(skyrl_entrypoint pid=546644)[0m max_output_tokens: 4096
|
||
[36m(skyrl_entrypoint pid=546644)[0m archiving:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: false
|
||
[36m(skyrl_entrypoint pid=546644)[0m trace_upload:
|
||
[36m(skyrl_entrypoint pid=546644)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m repo_org: DCAgent
|
||
[36m(skyrl_entrypoint pid=546644)[0m episodes: last
|
||
[36m(skyrl_entrypoint pid=546644)[0m dataset_type: SFT
|
||
[36m(skyrl_entrypoint pid=546644)[0m cleanup: true
|
||
[36m(skyrl_entrypoint pid=546644)[0m [0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(skyrl_entrypoint pid=546644)[0m No module named 'vllm._version'
|
||
[36m(skyrl_entrypoint pid=546644)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[36m(skyrl_entrypoint pid=546644)[0m W0608 20:41:39.902000 546644 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
|
||
[36m(pid=530352)[0m [2026-06-08 20:41:37,547 E 530352 544574] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[33m(raylet, ip=10.128.32.57)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 43x across cluster][0m
|
||
[36m(pid=2689828, ip=10.128.32.54)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(pid=2689828, ip=10.128.32.54)[0m No module named 'vllm._version'
|
||
[36m(pid=2689828, ip=10.128.32.54)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[33m(raylet, ip=10.128.32.58)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 109x across cluster][0m
|
||
[2026-06-08 20:41:53,824 E 546292 546616] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(pid=321222, ip=10.128.32.56)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 15x across cluster][0m
|
||
[36m(pid=321222, ip=10.128.32.56)[0m No module named 'vllm._version'[32m [repeated 15x across cluster][0m
|
||
[36m(pid=321222, ip=10.128.32.56)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 15x across cluster][0m
|
||
[36m(RegistryActor pid=2689760, ip=10.128.32.54)[0m [2026-06-08 20:41:54,703 E 2689760 2689800] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[33m(raylet, ip=10.128.32.55)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 71x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:41:59.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:41:59.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:41:59.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
|
||
[36m(pid=2794792, ip=10.128.32.52)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 9x across cluster][0m
|
||
[36m(pid=2794792, ip=10.128.32.52)[0m No module named 'vllm._version'[32m [repeated 9x across cluster][0m
|
||
[36m(pid=2794792, ip=10.128.32.52)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 9x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [2026-06-08 20:41:56,879 E 546644 546687] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:42:00.144 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:42:00.168 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:42:00.168 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:42:00.168 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:42:00.168 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m 2026-06-08 20:42:00.190 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.00s (attempt 1/5) to avoid port collisions
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[33m(raylet, ip=10.128.32.61)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 128x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [2026-06-08 20:42:04] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:03.798 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:03.799 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:03.799 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m No module named 'vllm._version'[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 19x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [2026-06-08 20:42:04] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
|
||
[36m(skyrl_entrypoint pid=546644)[0m [2026-06-08 20:42:05] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.367[0m | [1mINFO [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m_spin_up_http_endpoint[0m:[36m969[0m - [1mInferenceEngineClient HTTP endpoint started on 127.0.0.1:8000[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.367[0m | [1mINFO [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m__init__[0m:[36m61[0m - [1mInferenceEngineClient initialized with 48 engines.[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.369[0m | [1mINFO [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m_configure_harbor_logging[0m:[36m242[0m - [1mHarbor logging level set to INFO[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.370[0m | [1mINFO [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m__init__[0m:[36m142[0m - [1mTerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.371[0m | [1mINFO [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m__init__[0m:[36m158[0m - [1mTerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.372[0m | [1mINFO [0m | [36mskyrl_train.utils.trainer_utils[0m:[36mbuild_dataloader[0m:[36m656[0m - [1mTotal steps: 156[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.372[0m | [1mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m357[0m - [1mLength of train_dataloader: 5000[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.372[0m | [1mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m358[0m - [1mNumber of steps per epoch: 78[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:05.372[0m | [1mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m359[0m - [1mTotal training steps: 80[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:05.012 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:05.034 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:05.034 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:05.034 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:05.034 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m 2026-06-08 20:42:05.061 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 1.75s (attempt 1/5) to avoid port collisions[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 45x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m 2026-06-08 20:42:09.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m 2026-06-08 20:42:09.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m 2026-06-08 20:42:09.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m No module named 'vllm._version'[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 18x across cluster][0m
|
||
[36m(pid=3886563, ip=10.128.32.62)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m 2026-06-08 20:42:07.994 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m 2026-06-08 20:42:08.016 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m 2026-06-08 20:42:08.016 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m 2026-06-08 20:42:08.017 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m 2026-06-08 20:42:08.017 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m 2026-06-08 20:42:08.046 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.17s (attempt 1/5) to avoid port collisions[32m [repeated 8x across cluster][0m
|
||
[36m(bundle_reservation_check_func pid=546719)[0m [2026-06-08 20:42:12,634 E 546719 546759] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:10.747 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:10.748 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:10.748 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55)[0m No module named 'vllm._version'[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:12.143 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:12.166 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:12.166 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:12.167 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:12.167 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m 2026-06-08 20:42:12.200 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.09s (attempt 1/5) to avoid port collisions[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730194) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730194) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53)[0m [2026-06-08 20:42:16,474 E 2729834 2729931] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2729961, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730198) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690206) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m 2026-06-08 20:42:18.145 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m 2026-06-08 20:42:18.165 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m 2026-06-08 20:42:18.165 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m No module named 'vllm._version'[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690189) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870167) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730214) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869809, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870171) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m [W608 20:42:20.803504224 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-06.jupiter.internal]:38653 (errno: 97 - Address family not supported by protocol).
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m [W608 20:42:20.805060772 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m [rank0]:[W608 20:42:20.810164201 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690201) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730190) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m 2026-06-08 20:42:19.386 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61)[0m 2026-06-08 20:42:19.409 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61)[0m 2026-06-08 20:42:19.409 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61)[0m 2026-06-08 20:42:19.409 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61)[0m 2026-06-08 20:42:19.409 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m 2026-06-08 20:42:19.436 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.66s (attempt 1/5) to avoid port collisions[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321462) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870181) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=321221, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321449) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m 2026-06-08 20:42:21 INFO [ipv4-debug] hostname=jpbo-042-14.jupiter.internal
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m 2026-06-08 20:42:21 INFO [ipv4-debug] _global_node.node_ip_address=10.128.32.62
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m 2026-06-08 20:42:21 INFO [ipv4-debug] get_node_ip_address()=10.128.32.62
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948025) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176) _C._set_float32_matmul_precision(precision)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:22.012[0m | [1m[32mINFO [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m636[0m - [1m[32mInitializing process group for RayActorGroup[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690189)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=321220, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321469) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948034) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948034) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 17x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m [2026-06-08 20:42:21,583 E 321222 321386] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 32x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 74x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948043) _C._set_float32_matmul_precision(precision)
|
||
[36m(pid=3886639, ip=10.128.32.62)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948029) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999262) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m No module named 'vllm._version'[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m [W608 20:42:24.383004564 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-10.jupiter.internal]:42977 (errno: 97 - Address family not supported by protocol).[32m [repeated 20x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m [W608 20:42:24.383500619 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m [rank0]:[W608 20:42:24.385697605 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794924, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795150) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999273) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795159) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999277) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730190)
|
||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:13, 4.40s/it]
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795174) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948025)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794792, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795155) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794792, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795155) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m [2026-06-08 20:42:27,451 E 2998900 2999003] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120155, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120390) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120402) _C._set_float32_matmul_precision(precision)
|
||
[36m(pid=546824)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=3886641, ip=10.128.32.62)[0m [W608 20:42:30.447721940 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-14.jupiter.internal]:42967 (errno: 97 - Address family not supported by protocol).[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886641, ip=10.128.32.62)[0m [W608 20:42:30.448104973 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m [rank0]:[W608 20:42:29.576075878 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768833) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003938) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457)
|
||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:07<00:07, 3.89s/it][32m [repeated 35x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=768604, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768838) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893421) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916417) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003710, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003961) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003710, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003961) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59)[0m [2026-06-08 20:42:33,267 E 768606 768714] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 17x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:33.310[0m | [1m[32mINFO [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m638[0m - [1m[32mInitialized process group for RayActorGroup[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:33.320[0m | [1m[32mINFO [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m640[0m - [1m[32mMesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)][0m
|
||
[36m(FSDPPolicyWorkerBase pid=546822)[0m `torch_dtype` is deprecated! Use `dtype` instead!
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893429) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003956) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=768605, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768849) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=546822)[0m
|
||
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
|
||
[36m(FSDPPolicyWorkerBase pid=546822)[0m
|
||
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:00, 9.73it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=4003580, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003942) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916423) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768842) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893442) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=546822)[0m
|
||
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 2.95it/s]
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 4.51it/s]
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 4.95it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916413) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m [W608 20:42:35.989727680 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-13.jupiter.internal]:59347 (errno: 97 - Address family not supported by protocol).[32m [repeated 21x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m [W608 20:42:35.990281111 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m [rank0]:[W608 20:42:35.992209591 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185)
|
||
[36m(AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690201)
|
||
[36m(AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690189)
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690206)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730190)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729961, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730198)
|
||
[36m(AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730194)
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730214)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869809, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870171)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870181)
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870167)
|
||
[36m(AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321462)
|
||
[36m(AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321462)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:12<00:00, 3.22s/it][32m [repeated 66x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321221, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321449)
|
||
[36m(AsyncVLLMInferenceEngine pid=321220, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321469)
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948029)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 9x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [2026-06-08 20:42:36,503 E 3886563 3886603] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 8x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62)[0m `torch_dtype` is deprecated! Use `dtype` instead![32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62)[0m
|
||
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s][32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62)[0m
|
||
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 3.54it/s][32m [repeated 15x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948043)
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948034)
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948025)
|
||
[36m(FSDPPolicyWorkerBase pid=546824)[0m
|
||
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 2.95it/s]
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 4.49it/s][32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62)[0m
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 4.94it/s][32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690206) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m 2026-06-08 20:42:40.983 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999273)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999277)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999262)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794792, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795155)
|
||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.23s/it][32m [repeated 59x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768842)
|
||
[36m(AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768833)
|
||
[36m(AsyncVLLMInferenceEngine pid=768604, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768838)
|
||
[36m(AsyncVLLMInferenceEngine pid=768605, ip=10.128.32.59)[0m (EngineCore_DP0 pid=768849)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794792, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795155)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795174)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794924, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795150)
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m (EngineCore_DP0 pid=2795159)
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120402)
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999262) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386)
|
||
[36m(AsyncVLLMInferenceEngine pid=3120155, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120390)
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407)
|
||
[36m(FSDPPolicyWorkerBase pid=546822)[0m [rank6]:[W608 20:42:46.478528610 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m 2026-06-08 20:42:45.205 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00, 3.56s/it][32m [repeated 43x across cluster][0m
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:48.797[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mbuild_models[0m:[36m851[0m - [1m[32minit policy/ref/critic models done[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:48.802[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_maybe_enable_rollout_fanout[0m:[36m456[0m - [1m[32mRollout fan-out ENABLED: replacing single-process generator with RolloutDispatcher (K=4, cpus_per_coordinator=8).[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:48.824[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36m__init__[0m:[36m387[0m - [1m[32m[RolloutDispatcher] fan-out path: overriding inference host 127.0.0.1 -> 10.128.32.49 (routable head IP) for coordinator litellm base_url connectivity[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:48.824[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36m__init__[0m:[36m405[0m - [1m[32m[RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:48.829[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m434[0m - [1m[32m[RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893429)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893193, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893421)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893442)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003580, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003942)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003710, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003961)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003956)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m (EngineCore_DP0 pid=4003938)
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916423)
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916417)
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916413)
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405)
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893429) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 13x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62)[0m [rank1]:[W608 20:42:46.956187698 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 7x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m 2026-06-08 20:42:49.702 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.45s/it][32m [repeated 36x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=546825)[0m [2026-06-08 20:42:52,793 E 546825 546973] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.371 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 5x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:42:54.609[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 1/4 started[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.604 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.32.49:8000)
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.608 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.608 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:42:54.608 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete
|
||
[36m(RolloutCoordinator pid=547199)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m 2026-06-08 20:42:53.595 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 12x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=546824)[0m [2026-06-08 20:42:52,896 E 546824 547051] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 6x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:01.985 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[33m(raylet, ip=10.128.32.56)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.202 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.32.49:8000)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.205 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.205 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:43:02.205 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:02.205[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 2/4 started[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.597 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[33m(raylet, ip=10.128.32.54)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:10.821[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 3/4 started[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.817 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.817 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.817 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.818 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.32.49:8000)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.820 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.820 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:43:10.820 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.562 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[33m(raylet, ip=10.128.32.50)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.782[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 4/4 started[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.782[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m477[0m - [1m[32m[RolloutDispatcher] 4 coordinators started[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.782[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36mtrain[0m:[36m485[0m - [1m[32mGenerator startup complete[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.782[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m510[0m - [1m[32mStarted: 'load_checkpoints'[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.788[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1735[0m - [1m[32mLoading checkpoint from: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.788[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1741[0m - [1m[32mResuming from global_step: 83[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.794[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1757[0m - [1m[32mSuccessfully loaded trainer state[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.796[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1767[0m - [1m[32mSuccessfully loaded dataloader state[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:18.796[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1776[0m - [1m[32mLoading policy checkpoint from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.778 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.32.49:8000)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.781 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.781 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:43:18.781 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete
|
||
[36m(RolloutCoordinator pid=547199)[0m [2026-06-08 20:43:19,778 E 547199 547263] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [2026-06-08 20:43:27,452 E 321702 321742] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [2026-06-08 20:43:35,103 E 2690439 2690479] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m warnings.warn( # warn only once
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.246[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1786[0m - [1m[32mSuccessfully loaded policy checkpoint[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.246[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1802[0m - [1m[32mSuccessfully loaded complete checkpoint state from global_step_83[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.247[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m512[0m - [1m[32mResumed training from global_step 83[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.247[0m | [33m[1mWARNING [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m517[0m - [33m[1mNo data consumption state found in checkpoint — resume may re-train on already-consumed data[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.248[0m | [33m[1mWARNING [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m537[0m - [33m[1mData consumption count mismatch on resume: expected 320, got 0. This can happen after epoch boundary transitions or error recovery.[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.248[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m510[0m - [1m[32mFinished: 'load_checkpoints', time cost: 18.47s[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:37.248[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m544[0m - [1m[32mStarted: 'init_weight_sync_state'[0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m No module named 'vllm._version'
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [2026-06-08 20:43:43,749 E 3120643 3120685] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(FSDPPolicyWorkerBase pid=546824)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=546824)[0m No module named 'vllm._version'[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=546824)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [32m2026-06-08 20:43:43.980[0m | [1m[32mINFO [0m | [36mlogging[0m:[36minfo[0m:[36m2216[0m - [1m[32m[weight-sync] hostname=jpbo-042-14.jupiter.internal[0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [32m2026-06-08 20:43:43.980[0m | [1m[32mINFO [0m | [36mlogging[0m:[36minfo[0m:[36m2216[0m - [1m[32m[weight-sync] _global_node.node_ip_address=10.128.32.62[0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [32m2026-06-08 20:43:43.981[0m | [1m[32mINFO [0m | [36mlogging[0m:[36minfo[0m:[36m2216[0m - [1m[32m[weight-sync] get_node_ip_address()=10.128.32.62[0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [32m2026-06-08 20:43:43.981[0m | [1m[32mINFO [0m | [36mlogging[0m:[36minfo[0m:[36m2216[0m - [1m[32m[weight-sync] Using master_addr=10.128.32.62, master_port=56205[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916423) 2026-06-08 20:43:44.137 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:223 - torch.distributed.get_rank(): 0, rank_offset: 33, rank: 33, world_size: 49, group_name: skyrl
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m [rank0]:[W608 20:43:44.487817648 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-14.jupiter.internal]:56205 (errno: 97 - Address family not supported by protocol).
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916413) 2026-06-08 20:43:44.140 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:234 - init_weight_update_communicator: master_address=10.128.32.62, master_port=56205,
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m warnings.warn( # warn only once
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:44.211[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36minit_weight_sync_state[0m:[36m874[0m - [1m[32mInitialized weight sync state for policy model and inference engines.[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:44.212[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m544[0m - [1m[32mFinished: 'init_weight_sync_state', time cost: 6.96s[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56)[0m [rank0]:[W608 20:43:47.408195056 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:52.780[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m548[0m - [1m[32mFinished: 'sync_weights_to_inference_engines', time cost: 8.57s[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:52.781[0m | [1m[32mINFO [0m | [36mskyrl_train.callbacks.builtin[0m:[36mon_train_begin[0m:[36m270[0m - [1m[32mHFHubUploadCallback initialized: repo=laion/ablation-pymethods2test-seqnorm, upload_steps=5, export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Training Step Progress: 83it [00:00, ?it/s]
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 0%| | 0/64 [00:00<?, ?it/s][A[32m2026-06-08 20:43:52.783[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m607[0m - [1m[32mStarted: 'wait_for_generation_buffer'[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:43:52.783[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m604[0m - [1m[32mStarted: 'step'[0m[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457) 2026-06-08 20:43:44.134 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:223 - torch.distributed.get_rank(): 0, rank_offset: 12, rank: 12, world_size: 49, group_name: skyrl[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m [rank0]:[W608 20:43:44.602362089 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-042-14.jupiter.internal]:56205 (errno: 97 - Address family not supported by protocol).[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457) 2026-06-08 20:43:44.138 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:234 - init_weight_update_communicator: master_address=10.128.32.62, master_port=56205, [32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:43:52.786 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m [rank0]:[W608 20:43:47.875441937 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:43:58.155 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)[32m [repeated 171x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 2%|▏ | 1/64 [08:07<8:31:29, 487.13s/it][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 2%|▏ | 1/64 [08:07<8:31:29, 487.13s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:44:02.931 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)[32m [repeated 166x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:51:59.908 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 3%|▎ | 2/64 [09:35<4:21:02, 252.62s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 3%|▎ | 2/64 [09:35<4:21:02, 252.62s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:51:59.912 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:53:28.374 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:53:28.458 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 5%|▍ | 3/64 [09:43<2:23:04, 140.73s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 5%|▍ | 3/64 [09:43<2:23:04, 140.73s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:53:35.958 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:53:35.918 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:53:51.637 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 6%|▋ | 4/64 [09:58<1:31:22, 91.37s/it, buffer qsize=0] [A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 6%|▋ | 4/64 [09:58<1:31:22, 91.37s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:53:51.651 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:53:51.828 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 8%|▊ | 5/64 [09:59<57:30, 58.48s/it, buffer qsize=0] [A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 8%|▊ | 5/64 [09:59<57:30, 58.48s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 9%|▉ | 6/64 [10:10<41:00, 42.42s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 9%|▉ | 6/64 [10:10<41:00, 42.42s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:53:51.832 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:54:03.066 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:54:03.436 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 11%|█ | 7/64 [11:20<48:46, 51.35s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 11%|█ | 7/64 [11:20<48:46, 51.35s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=547199)[0m 2026-06-08 20:55:12.801 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m 2026-06-08 20:55:12.793 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 12%|█▎ | 8/64 [11:21<33:03, 35.41s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 12%|█▎ | 8/64 [11:21<33:03, 35.41s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 14%|█▍ | 9/64 [11:48<30:01, 32.76s/it, buffer qsize=0][A
|
||
[36m(skyrl_entrypoint pid=546644)[0m
|
||
Generation Buffer Progress: 14%|█▍ | 9/64 [11:48<30:01, 32.76s/it, buffer qsize=0][A
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:55:14.094 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:55:14.080 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m 2026-06-08 20:55:41.000 | INFO | examples.terminal_bench.terminal_bench_generator:generate:801 - Batch generation complete: 8/8 successful, 0 failed instances, 0 masked (excluded from baseline)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m 2026-06-08 20:55:41.005 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
slurmstepd: error: *** JOB 639476 ON jpbo-042-01 CANCELLED AT 2026-06-08T20:55:54 ***
|
||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ray_logs/
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:55:54.883[0m | [33m[1mWARNING [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m_mark_engine_dead[0m:[36m68[0m - [33m[1mInference engine 9 died (ActorDiedError). 47/48 engines remaining.[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:55:54.894[0m | [31m[1mERROR [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_run_generate_for_a_group_loop[0m:[36m961[0m - [31m[1mGenerator worker errored out with exception: The actor died unexpectedly before finishing this task.
|
||
[36m(skyrl_entrypoint pid=546644)[0m class_name: RolloutCoordinator
|
||
[36m(skyrl_entrypoint pid=546644)[0m actor_id: ab4de8c56e2033b1453f2d2002000000
|
||
[36m(skyrl_entrypoint pid=546644)[0m pid: 321702
|
||
[36m(skyrl_entrypoint pid=546644)[0m namespace: 9a22cdec-1723-4b53-8bb1-11eb041ab632
|
||
[36m(skyrl_entrypoint pid=546644)[0m ip: 10.128.32.56
|
||
[36m(skyrl_entrypoint pid=546644)[0m The actor died because its node has died. Node Id: 3e123e53cc2ba79145f3a0ef6259451f129cfaa5304440e6f5817516
|
||
[36m(skyrl_entrypoint pid=546644)[0m the actor's node was terminated expectedly: received SIGTERM[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [33m[1mTraceback (most recent call last):[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m File "[32m/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/[0m[32m[1mfully_async_trainer.py[0m", line [33m930[0m, in [35m_run_generate_for_a_group_loop[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [1mcur_generator_output[0m[1m:[0m [1mGeneratorOutput[0m [35m[1m=[0m [35m[1mawait[0m [1mself[0m[35m[1m.[0m[1mgenerator[0m[35m[1m.[0m[1mgenerate[0m[1m([0m[1mgenerator_input[0m[1m)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m File "[32m/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/[0m[32m[1mrollout_coordinator.py[0m", line [33m492[0m, in [35mgenerate[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [35m[1mreturn[0m [35m[1mawait[0m [1mactor[0m[35m[1m.[0m[1mrun_shard[0m[35m[1m.[0m[1mremote[0m[1m([0m[1minput_batch[0m[1m,[0m [1mglobal_step[0m[1m)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [31m[1mray.exceptions.ActorDiedError[0m:[1m The actor died unexpectedly before finishing this task.
|
||
[36m(skyrl_entrypoint pid=546644)[0m class_name: RolloutCoordinator
|
||
[36m(skyrl_entrypoint pid=546644)[0m actor_id: ab4de8c56e2033b1453f2d2002000000
|
||
[36m(skyrl_entrypoint pid=546644)[0m pid: 321702
|
||
[36m(skyrl_entrypoint pid=546644)[0m namespace: 9a22cdec-1723-4b53-8bb1-11eb041ab632
|
||
[36m(skyrl_entrypoint pid=546644)[0m ip: 10.128.32.56
|
||
[36m(skyrl_entrypoint pid=546644)[0m The actor died because its node has died. Node Id: 3e123e53cc2ba79145f3a0ef6259451f129cfaa5304440e6f5817516
|
||
[36m(skyrl_entrypoint pid=546644)[0m the actor's node was terminated expectedly: received SIGTERM[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [32m2026-06-08 20:55:54.951[0m | [33m[1mWARNING [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m_mark_engine_dead[0m:[36m68[0m - [33m[1mInference engine 8 died (ActorDiedError). 46/48 engines remaining.[0m
|
||
2026-06-08 20:55:55.045 | ERROR | __main__:main:134 - Training failed: Task failed because the node it was running on is dead or unavailable. Node IP: 10.128.32.49, node ID: 4cba01b979f7f003d04abf087b90b4e01890df9fe091283ac00c4962. This can happen if the node was preempted, had a hardware failure, or its raylet crashed unexpectedly. To see node death information, use `ray list nodes --filter node_id=4cba01b979f7f003d04abf087b90b4e01890df9fe091283ac00c4962`, check the Ray dashboard cluster page, search the node ID in the GCS logs, or use `ray logs raylet.out -ip 10.128.32.49`.
|
||
2026-06-08 20:55:55.045 | INFO | __main__:main:137 - Shutting down Ray on head node...
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:41:32] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:41:32] OK: RSS 1.38 GiB | node mem 148.6/858.0 GiB used (17.3%), avail 709.4 GiB
|
||
[36m(skyrl_entrypoint pid=546644)[0m ⚙️ Running in WANDB offline mode
|
||
[36m(skyrl_entrypoint pid=546644)[0m INFO 06-08 20:41:39 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
|
||
[36m(pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:41:57 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
|
||
[36m(pid=2689963, ip=10.128.32.54)[0m INFO 06-08 20:41:58 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m WARNING 06-08 20:42:02 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:42:02 [model.py:529] Resolved architecture: Qwen3ForCausalLM
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:42:02 [model.py:1549] Using max model len 32768
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:42:02 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:42:02 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:42:02 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m WARNING 06-08 20:42:02 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:42:02 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m WARNING 06-08 20:42:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m WARNING 06-08 20:42:02 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
|
||
[36m(pid=321222, ip=10.128.32.56)[0m INFO 06-08 20:42:03 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m WARNING 06-08 20:42:07 [arg_utils.py:1256] The global random seed is set to 49. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:42:07 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:42:07 [model.py:1549] Using max model len 32768[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:42:07 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:42:07 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:42:07 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m WARNING 06-08 20:42:07 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:42:07 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m WARNING 06-08 20:42:07 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m WARNING 06-08 20:42:07 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 11x across cluster][0m
|
||
[36m(pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=42, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m WARNING 06-08 20:42:10 [arg_utils.py:1256] The global random seed is set to 60. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:42:10 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:42:10 [model.py:1549] Using max model len 32768[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:42:10 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:42:10 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:42:10 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m WARNING 06-08 20:42:10 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:42:10 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m WARNING 06-08 20:42:13 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m WARNING 06-08 20:42:13 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 9x across cluster][0m
|
||
[36m(pid=4003708, ip=10.128.32.60)[0m INFO 06-08 20:42:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176) INFO 06-08 20:42:17 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=48, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m WARNING 06-08 20:42:15 [arg_utils.py:1256] The global random seed is set to 79. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:15 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:15 [model.py:1549] Using max model len 32768[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:15 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:15 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:15 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m WARNING 06-08 20:42:15 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:42:15 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730194) INFO 06-08 20:42:17 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m WARNING 06-08 20:42:15 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m WARNING 06-08 20:42:15 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m INFO 06-08 20:42:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 25x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:20 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.54:38653 backend=nccl
|
||
[36m(AsyncVLLMInferenceEngine pid=2729961, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730198) INFO 06-08 20:42:20 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:20 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
|
||
[36m(pid=3886563, ip=10.128.32.62)[0m ⚙️ Running in WANDB offline mode
|
||
[36m(AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690189) INFO 06-08 20:42:22 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:21 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=68, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m WARNING 06-08 20:42:22 [arg_utils.py:1256] The global random seed is set to 87. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:42:22 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:42:22 [model.py:1549] Using max model len 32768[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:42:22 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:42:22 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:42:22 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m WARNING 06-08 20:42:22 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:42:22 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948034) INFO 06-08 20:42:22 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 17x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m WARNING 06-08 20:42:22 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m WARNING 06-08 20:42:22 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:42:23 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 15x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:24 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.58:42977 backend=nccl[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:24 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:24 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m (EngineCore_DP0 pid=3948029) INFO 06-08 20:42:25 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120402) INFO 06-08 20:42:24 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=62, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:27 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 10x across cluster][0m
|
||
[36m(pid=3886639, ip=10.128.32.62)[0m ⚙️ Running in WANDB offline mode
|
||
[36m(AsyncVLLMInferenceEngine pid=768604, ip=10.128.32.59)[0m INFO 06-08 20:42:29 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:29 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.50:36953 backend=nccl[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:29 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:30 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:31 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:31 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=89, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893429) INFO 06-08 20:42:33 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 12x across cluster][0m
|
||
[36m(pid=546824)[0m ⚙️ Running in WANDB offline mode[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m INFO 06-08 20:42:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.61:59347 backend=nccl[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:36 [default_loader.py:293] Loading weights took 14.13 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m (EngineCore_DP0 pid=2690185) INFO 06-08 20:42:36 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.646981 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 16x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m (EngineCore_DP0 pid=3893434) INFO 06-08 20:42:35 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457) INFO 06-08 20:42:39 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457) INFO 06-08 20:42:39 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m (EngineCore_DP0 pid=321457) INFO 06-08 20:42:39 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730214) INFO 06-08 20:42:40 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=3869809, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870171) INFO 06-08 20:42:40 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.33 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176) WARNING 06-08 20:42:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176) INFO 06-08 20:42:40 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176) WARNING 06-08 20:42:40 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m (EngineCore_DP0 pid=3870176) INFO 06-08 20:42:40 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(AsyncVLLMInferenceEngine pid=2689961, ip=10.128.32.54)[0m WARNING 06-08 20:42:40 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999262) INFO 06-08 20:42:41 [default_loader.py:293] Loading weights took 13.38 seconds[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999262) INFO 06-08 20:42:41 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.428531 seconds[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:44 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999262) INFO 06-08 20:42:44 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) WARNING 06-08 20:42:45 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:45 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) WARNING 06-08 20:42:45 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m (EngineCore_DP0 pid=2999257) INFO 06-08 20:42:45 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m WARNING 06-08 20:42:45 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m NCCL version 2.27.7+cuda13.0
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:45 [default_loader.py:293] Loading weights took 14.23 seconds[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:46 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.346670 seconds[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:48 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:48 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:42:48 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:49 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.84 seconds[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) WARNING 06-08 20:42:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:49 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) WARNING 06-08 20:42:49 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120386) INFO 06-08 20:42:49 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m WARNING 06-08 20:42:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:49 [default_loader.py:293] Loading weights took 13.80 seconds[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:50 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.929588 seconds[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:42:54] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:42:54] OK: RSS 0.78 GiB | node mem 235.5/858.0 GiB used (27.5%), avail 622.4 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916417) INFO 06-08 20:42:52 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916417) INFO 06-08 20:42:52 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916417) INFO 06-08 20:42:52 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916413) INFO 06-08 20:42:52 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:52 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) WARNING 06-08 20:42:53 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:53 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) WARNING 06-08 20:42:53 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916172, ip=10.128.32.51)[0m (EngineCore_DP0 pid=2916405) INFO 06-08 20:42:53 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m WARNING 06-08 20:42:53 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:43:01] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:43:01] OK: RSS 0.78 GiB | node mem 399.2/858.0 GiB used (46.5%), avail 458.7 GiB
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:43:10] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:43:10] OK: RSS 0.78 GiB | node mem 377.6/858.0 GiB used (44.0%), avail 480.4 GiB
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:43:18] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:43:18] OK: RSS 0.78 GiB | node mem 415.1/858.0 GiB used (48.4%), avail 442.9 GiB
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy/model_world_size_8_rank_0.pt
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy/extra_state_world_size_8_rank_0.pt
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy/optim_world_size_8_rank_0.pt
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:43:32] OK: 102 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:43:32] OK: RSS 1.64 GiB | node mem 271.4/858.0 GiB used (31.6%), avail 586.6 GiB
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Successfully loaded model state dict
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Successfully loaded optimizer state
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Successfully loaded scheduler state
|
||
[36m(FSDPPolicyWorkerBase pid=3886563, ip=10.128.32.62)[0m [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints/global_step_83/policy
|
||
[36m(FSDPPolicyWorkerBase pid=546822)[0m INFO 06-08 20:43:40 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
|
||
[36m(AsyncVLLMInferenceEngine pid=2729834, ip=10.128.32.53)[0m (EngineCore_DP0 pid=2730190) INFO 06-08 20:43:52 [block_pool.py:452] Successfully reset prefix cache
|
||
[36m(FSDPPolicyWorkerBase pid=3886639, ip=10.128.32.62)[0m INFO 06-08 20:43:42 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 7x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m (EngineCore_DP0 pid=3120407) INFO 06-08 20:43:52 [block_pool.py:452] Successfully reset prefix cache[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 1512x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 503x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m INFO 06-08 20:44:42 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m INFO 06-08 20:44:42 [loggers.py:259] Engine 000: Avg prompt throughput: 8.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
||
[36m(AsyncVLLMInferenceEngine pid=3869940, ip=10.128.32.57)[0m INFO 06-08 20:44:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 45.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m INFO 06-08 20:44:48 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m INFO 06-08 20:44:48 [loggers.py:259] Engine 000: Avg prompt throughput: 8.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
||
[36m(AsyncVLLMInferenceEngine pid=4003580, ip=10.128.32.60)[0m INFO 06-08 20:44:52 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m INFO 06-08 20:44:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%[32m [repeated 4x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:44:54] OK: 215 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:44:54] OK: RSS 1.07 GiB | node mem 256.5/858.0 GiB used (29.9%), avail 601.5 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2689828, ip=10.128.32.54)[0m INFO 06-08 20:44:57 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m INFO 06-08 20:44:58 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794792, ip=10.128.32.52)[0m INFO 06-08 20:44:59 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:45:01] OK: 330 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:45:01] OK: RSS 1.07 GiB | node mem 407.9/858.0 GiB used (47.5%), avail 450.1 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m INFO 06-08 20:45:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 7x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m INFO 06-08 20:45:04 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 4x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:45:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%[32m [repeated 9x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:45:10] OK: 393 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:45:10] OK: RSS 1.11 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.7 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=3869941, ip=10.128.32.57)[0m INFO 06-08 20:45:10 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916174, ip=10.128.32.51)[0m INFO 06-08 20:45:13 [loggers.py:259] Engine 000: Avg prompt throughput: 8.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768605, ip=10.128.32.59)[0m INFO 06-08 20:45:15 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:45:18] OK: 403 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:45:18] OK: RSS 1.12 GiB | node mem 423.5/858.0 GiB used (49.4%), avail 434.4 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m INFO 06-08 20:45:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54)[0m INFO 06-08 20:45:19 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:45:22 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m INFO 06-08 20:45:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:45:26 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m INFO 06-08 20:45:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 2x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:45:32] OK: 135 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:45:32] OK: RSS 1.75 GiB | node mem 256.6/858.0 GiB used (29.9%), avail 601.4 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2916039, ip=10.128.32.51)[0m INFO 06-08 20:45:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54)[0m INFO 06-08 20:45:39 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:45:45 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 53.2%[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768604, ip=10.128.32.59)[0m INFO 06-08 20:45:49 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:45:51 [loggers.py:259] Engine 000: Avg prompt throughput: 153.9 tokens/s, Avg generation throughput: 74.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 56.2%[32m [repeated 17x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794924, ip=10.128.32.52)[0m INFO 06-08 20:45:53 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:45:56 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 86.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 56.2%[32m [repeated 20x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947669, ip=10.128.32.58)[0m INFO 06-08 20:45:59 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:46:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 86.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 56.2%[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003710, ip=10.128.32.60)[0m INFO 06-08 20:46:04 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:46:06 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 56.2%[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794925, ip=10.128.32.52)[0m INFO 06-08 20:46:06 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:46:11 [loggers.py:259] Engine 000: Avg prompt throughput: 76.6 tokens/s, Avg generation throughput: 55.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 57.2%[32m [repeated 27x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999029, ip=10.128.32.55)[0m INFO 06-08 20:46:11 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 4x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689962, ip=10.128.32.54)[0m INFO 06-08 20:46:15 [loggers.py:259] Engine 000: Avg prompt throughput: 24.7 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 32.8%[32m [repeated 27x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:46:16 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:46:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 225.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 66.7%[32m [repeated 31x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59)[0m INFO 06-08 20:46:21 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120027, ip=10.128.32.50)[0m INFO 06-08 20:46:26 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 191.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 66.7%[32m [repeated 31x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2729962, ip=10.128.32.53)[0m INFO 06-08 20:46:29 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:46:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 36x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893192, ip=10.128.32.61)[0m INFO 06-08 20:46:32 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:46:36 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%[32m [repeated 36x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:46:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%[32m [repeated 33x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:46:46 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%[32m [repeated 35x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:46:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%[32m [repeated 37x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999155, ip=10.128.32.55)[0m INFO 06-08 20:46:53 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:46:54] OK: 256 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:46:54] OK: RSS 1.11 GiB | node mem 256.6/858.0 GiB used (29.9%), avail 601.4 GiB
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:46:56 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%[32m [repeated 38x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003708, ip=10.128.32.60)[0m INFO 06-08 20:46:55 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:47:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%[32m [repeated 36x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:47:01] OK: 260 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:47:01] OK: RSS 1.12 GiB | node mem 408.0/858.0 GiB used (47.5%), avail 450.0 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m INFO 06-08 20:46:59 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:47:06 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%[32m [repeated 38x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 3x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:47:10] OK: 388 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:47:10] OK: RSS 1.13 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.7 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:47:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%[32m [repeated 38x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=4003709, ip=10.128.32.60)[0m INFO 06-08 20:47:16 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%[32m [repeated 36x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:47:18] OK: 404 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:47:18] OK: RSS 1.13 GiB | node mem 423.8/858.0 GiB used (49.4%), avail 434.2 GiB
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2999030, ip=10.128.32.55)[0m INFO 06-08 20:47:21 [loggers.py:259] Engine 000: Avg prompt throughput: 88.4 tokens/s, Avg generation throughput: 30.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 76.0%[32m [repeated 39x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768606, ip=10.128.32.59)[0m INFO 06-08 20:47:26 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 188.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 69.0%[32m [repeated 43x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:47:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 46.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 63.1%[32m [repeated 38x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:47:32] OK: 228 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:47:32] OK: RSS 1.76 GiB | node mem 256.6/858.0 GiB used (29.9%), avail 601.4 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:47:36 [loggers.py:259] Engine 000: Avg prompt throughput: 153.0 tokens/s, Avg generation throughput: 82.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 67.6%[32m [repeated 40x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:47:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 92.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 67.6%[32m [repeated 38x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947798, ip=10.128.32.58)[0m INFO 06-08 20:47:43 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:47:46 [loggers.py:259] Engine 000: Avg prompt throughput: 75.9 tokens/s, Avg generation throughput: 70.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 70.5%[32m [repeated 37x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:47:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 91.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 70.5%[32m [repeated 39x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3120157, ip=10.128.32.50)[0m INFO 06-08 20:47:54 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:47:56 [loggers.py:259] Engine 000: Avg prompt throughput: 140.2 tokens/s, Avg generation throughput: 108.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 73.9%[32m [repeated 42x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 134.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 73.9%[32m [repeated 42x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 3x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:06 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 90.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 73.9%[32m [repeated 41x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 7x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 90.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 73.9%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:48:16 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=3120156, ip=10.128.32.50)[0m INFO 06-08 20:48:16 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 84.6%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 20x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:21 [loggers.py:259] Engine 000: Avg prompt throughput: 119.5 tokens/s, Avg generation throughput: 166.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 75.5%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:26 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 136.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 75.5%[32m [repeated 42x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 6x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 129.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 75.5%[32m [repeated 40x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 27x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:37 [loggers.py:259] Engine 000: Avg prompt throughput: 76.6 tokens/s, Avg generation throughput: 170.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 78.4%[32m [repeated 41x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321092, ip=10.128.32.56)[0m INFO 06-08 20:48:40 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 78.4%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:47 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 78.4%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:52 [loggers.py:259] Engine 000: Avg prompt throughput: 72.9 tokens/s, Avg generation throughput: 199.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 78.5%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:48:54] OK: 284 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:48:54] OK: RSS 1.14 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 16x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:48:57 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 78.5%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 19x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:49:01] OK: 383 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:49:01] OK: RSS 1.16 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 450.0 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 138.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 78.5%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:07 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 116.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 78.5%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 8x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:49:10] OK: 345 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:49:10] OK: RSS 1.14 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.7 GiB
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2689963, ip=10.128.32.54)[0m INFO 06-08 20:49:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 138.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 84.1%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:17 [loggers.py:259] Engine 000: Avg prompt throughput: 67.2 tokens/s, Avg generation throughput: 122.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 78.8%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:49:18] OK: 264 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:49:18] OK: RSS 1.14 GiB | node mem 423.8/858.0 GiB used (49.4%), avail 434.1 GiB
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 8x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 96.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 78.8%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:27 [loggers.py:259] Engine 000: Avg prompt throughput: 113.4 tokens/s, Avg generation throughput: 63.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 78.9%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:32 [loggers.py:259] Engine 000: Avg prompt throughput: 33.9 tokens/s, Avg generation throughput: 97.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 79.8%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:49:32] OK: 287 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:49:32] OK: RSS 1.77 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 21x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 79.8%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:42 [loggers.py:259] Engine 000: Avg prompt throughput: 22.7 tokens/s, Avg generation throughput: 50.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 80.8%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947797, ip=10.128.32.58)[0m INFO 06-08 20:49:47 [loggers.py:259] Engine 000: Avg prompt throughput: 210.5 tokens/s, Avg generation throughput: 117.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 82.3%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:49:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 75.6%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 6x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:49:57 [loggers.py:259] Engine 000: Avg prompt throughput: 122.0 tokens/s, Avg generation throughput: 50.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 80.2%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 14x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 80.2%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:07 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 80.2%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 22x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:12 [loggers.py:259] Engine 000: Avg prompt throughput: 23.2 tokens/s, Avg generation throughput: 77.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 85.4%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=321222, ip=10.128.32.56)[0m INFO 06-08 20:50:13 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 14x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:17 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 94.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 85.4%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 94.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 85.4%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 19x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:27 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 85.4%[32m [repeated 44x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 23x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:32 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 85.4%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 25x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 85.4%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 17x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 85.4%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:47 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 85.4%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 7x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3869939, ip=10.128.32.57)[0m INFO 06-08 20:50:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 85.4%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:50:54] OK: 273 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:50:54] OK: RSS 1.15 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:50:57 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 123.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 85.8%[32m [repeated 46x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 11x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:51:01] OK: 399 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:51:01] OK: RSS 1.16 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 450.0 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 92.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 85.8%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:07 [loggers.py:259] Engine 000: Avg prompt throughput: 136.6 tokens/s, Avg generation throughput: 135.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 85.7%[32m [repeated 45x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:51:10] OK: 361 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:51:10] OK: RSS 1.15 GiB | node mem 386.3/858.0 GiB used (45.0%), avail 471.6 GiB
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 4x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:12 [loggers.py:259] Engine 000: Avg prompt throughput: 42.8 tokens/s, Avg generation throughput: 164.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.4%, Prefix cache hit rate: 85.9%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:17 [loggers.py:259] Engine 000: Avg prompt throughput: 99.3 tokens/s, Avg generation throughput: 216.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 85.5%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:51:18] OK: 276 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:51:18] OK: RSS 1.15 GiB | node mem 423.9/858.0 GiB used (49.4%), avail 434.1 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 206.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.4%, Prefix cache hit rate: 85.5%[32m [repeated 49x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:27 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 85.5%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:51:32] OK: 332 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:51:32] OK: RSS 1.78 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.2 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:32 [loggers.py:259] Engine 000: Avg prompt throughput: 92.1 tokens/s, Avg generation throughput: 219.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.1%, Prefix cache hit rate: 85.1%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 198.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.0%, Prefix cache hit rate: 85.1%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:42 [loggers.py:259] Engine 000: Avg prompt throughput: 209.6 tokens/s, Avg generation throughput: 201.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 84.9%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 8x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 258.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.5%, Prefix cache hit rate: 84.9%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 17x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:53 [loggers.py:259] Engine 000: Avg prompt throughput: 23.6 tokens/s, Avg generation throughput: 225.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 85.4%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 11x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:51:58 [loggers.py:259] Engine 000: Avg prompt throughput: 35.0 tokens/s, Avg generation throughput: 236.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 85.6%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 25x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:52:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 191.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.3%, Prefix cache hit rate: 85.6%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:52:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 85.6%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 30x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:52:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 155.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.1%, Prefix cache hit rate: 85.6%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 19x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:52:18 [loggers.py:259] Engine 000: Avg prompt throughput: 235.2 tokens/s, Avg generation throughput: 169.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 84.9%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 25x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:52:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 141.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 84.9%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 23x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2794923, ip=10.128.32.52)[0m INFO 06-08 20:52:28 [loggers.py:259] Engine 000: Avg prompt throughput: 165.8 tokens/s, Avg generation throughput: 164.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 85.3%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m INFO 06-08 20:52:33 [loggers.py:259] Engine 000: Avg prompt throughput: 27.5 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 86.3%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m INFO 06-08 20:52:38 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 91.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 86.3%[32m [repeated 47x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3947796, ip=10.128.32.58)[0m INFO 06-08 20:52:43 [loggers.py:259] Engine 000: Avg prompt throughput: 20.9 tokens/s, Avg generation throughput: 91.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 86.6%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 28x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59)[0m INFO 06-08 20:52:48 [loggers.py:259] Engine 000: Avg prompt throughput: 13.9 tokens/s, Avg generation throughput: 213.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 93.7%[32m [repeated 49x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59)[0m INFO 06-08 20:52:53 [loggers.py:259] Engine 000: Avg prompt throughput: 9.6 tokens/s, Avg generation throughput: 163.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.4%, Prefix cache hit rate: 93.9%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:52:54] OK: 266 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:52:54] OK: RSS 1.15 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 19x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59)[0m INFO 06-08 20:52:58 [loggers.py:259] Engine 000: Avg prompt throughput: 9.6 tokens/s, Avg generation throughput: 148.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 94.1%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 29x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:53:01] OK: 408 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:53:01] OK: RSS 1.17 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 450.0 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=768476, ip=10.128.32.59)[0m INFO 06-08 20:53:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 179.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 94.1%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m INFO 06-08 20:53:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 253.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 87.7%[32m [repeated 49x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:53:10] OK: 379 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:53:10] OK: RSS 1.19 GiB | node mem 386.4/858.0 GiB used (45.0%), avail 471.6 GiB
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m INFO 06-08 20:53:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 87.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 8x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m INFO 06-08 20:53:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 87.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:53:18] OK: 275 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:53:18] OK: RSS 1.15 GiB | node mem 423.9/858.0 GiB used (49.4%), avail 434.1 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893064, ip=10.128.32.61)[0m INFO 06-08 20:53:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 87.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:28 [loggers.py:259] Engine 000: Avg prompt throughput: 60.3 tokens/s, Avg generation throughput: 165.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 82.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:53:32] OK: 339 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:53:32] OK: RSS 1.79 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.3 GiB
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:33 [loggers.py:259] Engine 000: Avg prompt throughput: 63.3 tokens/s, Avg generation throughput: 191.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.0%, Prefix cache hit rate: 82.6%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 36x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:38 [loggers.py:259] Engine 000: Avg prompt throughput: 56.1 tokens/s, Avg generation throughput: 196.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 83.8%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:43 [loggers.py:259] Engine 000: Avg prompt throughput: 187.6 tokens/s, Avg generation throughput: 233.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 41x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 205.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 20x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 170.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 84.5%[32m [repeated 49x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 17x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:53:58 [loggers.py:259] Engine 000: Avg prompt throughput: 64.9 tokens/s, Avg generation throughput: 191.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 213.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:08 [loggers.py:259] Engine 000: Avg prompt throughput: 52.8 tokens/s, Avg generation throughput: 254.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 31x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 217.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 20x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:18 [loggers.py:259] Engine 000: Avg prompt throughput: 106.0 tokens/s, Avg generation throughput: 193.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 15x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 213.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 13x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 213.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:54:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 284.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 85.8%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:54:38 [loggers.py:259] Engine 000: Avg prompt throughput: 33.5 tokens/s, Avg generation throughput: 317.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 85.8%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:54:43 [loggers.py:259] Engine 000: Avg prompt throughput: 68.9 tokens/s, Avg generation throughput: 367.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 85.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 25x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:54:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 378.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 85.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 31x across cluster][0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:54:53 [loggers.py:259] Engine 000: Avg prompt throughput: 60.8 tokens/s, Avg generation throughput: 398.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 85.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:54:54] OK: 299 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=547199)[0m [fd-monitor] [20:54:54] OK: RSS 1.20 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.2 GiB
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 2x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:54:58 [loggers.py:259] Engine 000: Avg prompt throughput: 73.8 tokens/s, Avg generation throughput: 305.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 8x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:55:01] OK: 431 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [fd-monitor] [20:55:01] OK: RSS 1.21 GiB | node mem 408.0/858.0 GiB used (47.6%), avail 449.9 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:55:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 297.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 7x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:55:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 298.0 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:55:10] OK: 407 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [fd-monitor] [20:55:10] OK: RSS 1.20 GiB | node mem 386.4/858.0 GiB used (45.0%), avail 471.6 GiB
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 28x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:55:13 [loggers.py:259] Engine 000: Avg prompt throughput: 45.8 tokens/s, Avg generation throughput: 305.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.6%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 12x across cluster][0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=547199)[0m
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:55:18] OK: 414 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=3120643, ip=10.128.32.50)[0m [fd-monitor] [20:55:18] OK: RSS 1.20 GiB | node mem 423.8/858.0 GiB used (49.4%), avail 434.1 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:55:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 310.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 10x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2998900, ip=10.128.32.55)[0m INFO 06-08 20:55:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 299.4 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 84.5%[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:55:28 [loggers.py:259] Engine 000: Avg prompt throughput: 198.0 tokens/s, Avg generation throughput: 404.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.2%, Prefix cache hit rate: 85.2%[32m [repeated 49x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:55:32] OK: 458 / 131,072 FDs open (0.3% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=546644)[0m [fd-monitor] [20:55:32] OK: RSS 1.80 GiB | node mem 256.7/858.0 GiB used (29.9%), avail 601.2 GiB
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:55:33 [loggers.py:259] Engine 000: Avg prompt throughput: 240.9 tokens/s, Avg generation throughput: 419.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.6%, Prefix cache hit rate: 84.8%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:55:38 [loggers.py:259] Engine 000: Avg prompt throughput: 236.9 tokens/s, Avg generation throughput: 396.2 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 84.9%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 5x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:55:43 [loggers.py:259] Engine 000: Avg prompt throughput: 87.2 tokens/s, Avg generation throughput: 425.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.6%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 11x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2730089, ip=10.128.32.53)[0m INFO 06-08 20:55:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 378.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 84.7%[32m [repeated 48x across cluster][0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m [1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new[0m
|
||
[36m(RolloutCoordinator pid=321702, ip=10.128.32.56)[0m LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
|
||
[36m(RolloutCoordinator pid=2690439, ip=10.128.32.54)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 7x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=3893191, ip=10.128.32.61)[0m INFO 06-08 20:55:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 87.7%[32m [repeated 49x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=2916173, ip=10.128.32.51)[0m INFO 06-08 20:55:54 [loggers.py:259] Engine 000: Avg prompt throughput: 552.6 tokens/s, Avg generation throughput: 360.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 85.9%[32m [repeated 4x across cluster][0m
|
||
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqnorm', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqnorm', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
|
||
Traceback (most recent call last):
|
||
File "<frozen runpy>", line 198, in _run_module_as_main
|
||
File "<frozen runpy>", line 88, in _run_code
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
|
||
main()
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
|
||
_run_hydra(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
|
||
_run_app(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
|
||
run_and_report(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
|
||
raise ex
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
|
||
return func()
|
||
^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
|
||
lambda: hydra.run(
|
||
^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
|
||
_ = ret.return_value
|
||
^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
|
||
raise self._return_value
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
|
||
ret.return_value = task_function(task_cfg)
|
||
^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
|
||
ray.get(skyrl_entrypoint.remote(cfg))
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
|
||
return fn(*args, **kwargs)
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
|
||
return func(*args, **kwargs)
|
||
^^^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
|
||
values, debugger_breakpoint = worker.get_objects(
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1017, in get_objects
|
||
raise value
|
||
ray.exceptions.NodeDiedError: Task failed because the node it was running on is dead or unavailable. Node IP: 10.128.32.49, node ID: 4cba01b979f7f003d04abf087b90b4e01890df9fe091283ac00c4962. This can happen if the node was preempted, had a hardware failure, or its raylet crashed unexpectedly. To see node death information, use `ray list nodes --filter node_id=4cba01b979f7f003d04abf087b90b4e01890df9fe091283ac00c4962`, check the Ray dashboard cluster page, search the node ID in the GCS logs, or use `ray logs raylet.out -ip 10.128.32.49`.
|
||
Collecting Ray logs from worker jpbo-042-02...
|
||
WARNING: Failed to collect logs from jpbo-042-02
|
||
Collecting Ray logs from worker jpbo-042-03...
|
||
Stopping Ray cluster...
|
||
WARNING: Failed to collect logs from jpbo-042-03
|
||
Collecting Ray logs from worker jpbo-042-04...
|
||
WARNING: Failed to collect logs from jpbo-042-04
|
||
Collecting Ray logs from worker jpbo-042-05...
|
||
WARNING: Failed to collect logs from jpbo-042-05
|
||
Collecting Ray logs from worker jpbo-042-06...
|
||
Ray cluster stopped
|
||
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||
WARNING: Failed to collect logs from jpbo-042-06
|
||
Collecting Ray logs from worker jpbo-042-07...
|
||
Collecting Ray logs from worker jpbo-042-02...
|
||
WARNING: Failed to collect logs from jpbo-042-07
|
||
Collecting Ray logs from worker jpbo-042-08...
|
||
WARNING: Failed to collect logs from jpbo-042-02
|
||
Collecting Ray logs from worker jpbo-042-03...
|
||
WARNING: Failed to collect logs from jpbo-042-03
|
||
Collecting Ray logs from worker jpbo-042-04...
|
||
WARNING: Failed to collect logs from jpbo-042-04
|
||
Collecting Ray logs from worker jpbo-042-05...
|
||
WARNING: Failed to collect logs from jpbo-042-08
|
||
Collecting Ray logs from worker jpbo-042-09...
|
||
WARNING: Failed to collect logs from jpbo-042-05
|
||
Collecting Ray logs from worker jpbo-042-06...
|
||
WARNING: Failed to collect logs from jpbo-042-06
|
||
Collecting Ray logs from worker jpbo-042-07...
|
||
WARNING: Failed to collect logs from jpbo-042-07
|
||
Collecting Ray logs from worker jpbo-042-08...
|
||
WARNING: Failed to collect logs from jpbo-042-08
|
||
Collecting Ray logs from worker jpbo-042-09...
|
||
WARNING: Failed to collect logs from jpbo-042-09
|
||
Collecting Ray logs from worker jpbo-042-10...
|
||
WARNING: Failed to collect logs from jpbo-042-09
|
||
Collecting Ray logs from worker jpbo-042-10...
|
||
WARNING: Failed to collect logs from jpbo-042-10
|
||
WARNING: Failed to collect logs from jpbo-042-10
|
||
Collecting Ray logs from worker jpbo-042-11...
|
||
Collecting Ray logs from worker jpbo-042-11...
|
||
WARNING: Failed to collect logs from jpbo-042-11
|
||
Collecting Ray logs from worker jpbo-042-12...
|
||
WARNING: Failed to collect logs from jpbo-042-11
|
||
Collecting Ray logs from worker jpbo-042-12...
|
||
WARNING: Failed to collect logs from jpbo-042-12
|
||
Collecting Ray logs from worker jpbo-042-13...
|
||
WARNING: Failed to collect logs from jpbo-042-12
|
||
Collecting Ray logs from worker jpbo-042-13...
|
||
WARNING: Failed to collect logs from jpbo-042-13
|
||
Collecting Ray logs from worker jpbo-042-14...
|
||
WARNING: Failed to collect logs from jpbo-042-13
|
||
Collecting Ray logs from worker jpbo-042-14...
|
||
WARNING: Failed to collect logs from jpbo-042-14
|
||
Ray log preservation complete
|
||
WARNING: Failed to collect logs from jpbo-042-14
|
||
[RLJobRunner] Crash-time Ray log preservation complete.
|
||
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||
repo_id: DCAgent/ablation-pymethods2test-seqnorm
|
||
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/ablation-pymethods2test-seqnorm
|
||
episodes: last
|
||
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqnorm_3/logs/ablation-pymethods2test-seqnorm_trace_upload.log
|
||
[RLJobRunner] Waiting for trace upload to complete...
|