The following have been reloaded with a version change: 1) GCCcore/.14.3.0 => GCCcore/14.3.0 Lmod is automatically replacing "GCC/14.3.0" with "nvidia-compilers/25.9-CUDA-13". Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python [ray] RAY_TMPDIR=/tmp/ray/ray_736687 [triton_cache] Triton cache: /tmp/triton_cache_feuer1_736687 [triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_736687 [proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 [proxy] Setting up SSH tunnel to jpbl-s01-01 [proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc [proxy] Tunnel port: 7003 [proxy] Node IP: 10.128.25.17 (workers will connect here) [proxy] ✓ SSH tunnel started successfully [proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_736687.conf [proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT [proxy] - External traffic (internet) → PROXY via tunnel [proxy] ✓ Daytona timeout settings configured [proxy] Testing proxy connectivity... [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_736687.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary) [proxy] ✓ Tunnel accessible at 10.128.25.17:7003 (workers can connect) [proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers) [container_runtime] Using cloud backend: daytona (no local container setup) === Universal RL Training Runner === Config: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/configs/explore-tis-untrunc_rl_config.json Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python version: Python 3.12.12 UV_USE_IO_URING: 0 Proxy: DISABLED (direct internet or not configured) ======================================== === RLJobRunner: explore-tis-untrunc === [wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/wandb [wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/wandb HF_TOKEN=****pDbg HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config) Environment configured: TENSOR_PARALLEL_SIZE=1 NUM_INFERENCE_ENGINES=56 POLICY_NUM_NODES=14 WANDB_DIR=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/wandb Starting Ray cluster with 14 nodes, 4 GPUs/node Cleaning up existing Ray instances... === Starting Ray Cluster === Nodes: 14 GPUs per node: 4 CPUs per node: 288 Head node: jpbo-026-33 (10.128.25.17) Ray port: 6379 ============================ Starting Ray head on jpbo-026-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-026-33.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.17 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.25.17 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray head on jpbo-026-33 Starting Ray worker on jpbo-026-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-39.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.23 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.23 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 1 on jpbo-026-39 Starting Ray worker on jpbo-026-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-40.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.24 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.24 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 2 on jpbo-026-40 Starting Ray worker on jpbo-026-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-41.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.25 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.25 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 3 on jpbo-026-41 Starting Ray worker on jpbo-026-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-43.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.27 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.27 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 4 on jpbo-026-43 Starting Ray worker on jpbo-026-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-44.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.28 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.28 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 5 on jpbo-026-44 Starting Ray worker on jpbo-029-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-33.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.161 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.161 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 6 on jpbo-029-33 Starting Ray worker on jpbo-029-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-34.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.162 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.162 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 7 on jpbo-029-34 Starting Ray worker on jpbo-029-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-35.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.163 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.163 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 8 on jpbo-029-35 Starting Ray worker on jpbo-029-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-36.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.164 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.164 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 9 on jpbo-029-36 Starting Ray worker on jpbo-029-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-37.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.165 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.165 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 10 on jpbo-029-37 Starting Ray worker on jpbo-029-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-44.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.172 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.172 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 11 on jpbo-029-44 Starting Ray worker on jpbo-029-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-46.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.174 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.174 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 12 on jpbo-029-46 Starting Ray worker on jpbo-029-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-48.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.176 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.176 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 13 on jpbo-029-48 Waiting for cluster (56 GPUs, 14 nodes)... Connecting to Ray at 10.128.25.17:6379 (expecting 14 nodes, 56.0 GPUs) Ray connection established, polling for resources... [Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'CPU': 4032.0, 'object_store_memory': 601295421440.0, 'memory': 10860264161280.0, 'GPU': 56.0, 'accelerator_type:GH200': 14.0, 'node:10.128.25.161': 1.0, 'node:10.128.25.176': 1.0, 'node:10.128.25.162': 1.0, 'node:10.128.25.25': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.25.17': 1.0, 'node:10.128.25.23': 1.0, 'node:10.128.25.164': 1.0, 'node:10.128.25.165': 1.0, 'node:10.128.25.24': 1.0, 'node:10.128.25.172': 1.0, 'node:10.128.25.174': 1.0, 'node:10.128.25.28': 1.0, 'node:10.128.25.163': 1.0, 'node:10.128.25.27': 1.0} ✓ Ray cluster ready === Ray Cluster Ready === Address: 10.128.25.17:6379 Total GPUs: 56 ========================= Ray cluster ready at 10.128.25.17:6379 Total GPUs available: 56 [RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona) [RLJobRunner] No Pinggy tunnel needed, using local vLLM Running SkyRL: Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Entrypoint: examples.terminal_bench.entrypoints.main_tbench Args: 126 Hydra arguments Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global trainer.algorithm.use_tis=true trainer.algorithm.tis_imp_ratio_cap=2.0 trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/explore-tis-untrunc ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=explore-tis-untrunc trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.collect_rollout_details=true +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true generator.sampling_params.temperature=1.0 generator.sampling_params.top_p=1.0 generator.sampling_params.top_k=-1 [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_736687.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e 2026-06-12 00:53:52.012 | WARNING | skyrl_train.utils.utils:validate_cfg:479 - `generator.sampling_params.logprobs` is `None` but `trainer.algorithm.use_tis` is `True`. Setting `logprobs` to `True`. 2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:833 - Exporting wandb api key to ray runtime env 2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:852 - Exporting RAY_ADDRESS to ray runtime env 2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:877 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0 2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:877 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET 2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:877 - Exporting `NCCL_DEBUG` to ray runtime env: WARN 2026-06-12 00:53:52,180 INFO worker.py:1680 -- Using address 10.128.25.17:6379 set in the environment variable RAY_ADDRESS 2026-06-12 00:53:52,214 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.25.17:6379... 2026-06-12 00:53:52,222 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. 2026-06-12 00:53:52,225 INFO worker.py:2007 -- Connected to Ray cluster. /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (pid=348050) [2026-06-12 00:53:52,634 E 348050 349258] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet) 2026-06-12 00:53:53,525 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. 2026-06-12 00:53:54.651 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (pid=348068) [2026-06-12 00:53:58,380 E 348068 349758] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 8x across cluster] (raylet) 2026-06-12 00:53:55,365 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 2x across cluster] (skyrl_entrypoint pid=364638) 2026-06-12 00:54:01.116 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:214 - SkyRL log level set to: INFO (skyrl_entrypoint pid=364638) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (skyrl_entrypoint pid=364638) 2026-06-12 00:54:01.404 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large (skyrl_entrypoint pid=364638) 2026-06-12 00:54:04.195 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories (skyrl_entrypoint pid=364638) 2026-06-12 00:54:04.195 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths (skyrl_entrypoint pid=364638) 2026-06-12 00:54:04.206 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:407 - data: (skyrl_entrypoint pid=364638) train_data: (skyrl_entrypoint pid=364638) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large (skyrl_entrypoint pid=364638) val_data: [] (skyrl_entrypoint pid=364638) trainer: (skyrl_entrypoint pid=364638) placement: (skyrl_entrypoint pid=364638) colocate_all: false (skyrl_entrypoint pid=364638) colocate_policy_ref: true (skyrl_entrypoint pid=364638) policy_num_nodes: 2 (skyrl_entrypoint pid=364638) policy_num_gpus_per_node: 4 (skyrl_entrypoint pid=364638) critic_num_nodes: 1 (skyrl_entrypoint pid=364638) critic_num_gpus_per_node: 4 (skyrl_entrypoint pid=364638) ref_num_nodes: 2 (skyrl_entrypoint pid=364638) ref_num_gpus_per_node: 4 (skyrl_entrypoint pid=364638) policy_strict_spread_pg: false (skyrl_entrypoint pid=364638) policy_per_gpu_bundles: false (skyrl_entrypoint pid=364638) policy_force_cvd_mask: false (skyrl_entrypoint pid=364638) sequence_parallel_backend: ulysses (skyrl_entrypoint pid=364638) strategy: fsdp2 (skyrl_entrypoint pid=364638) policy: (skyrl_entrypoint pid=364638) model: (skyrl_entrypoint pid=364638) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 (skyrl_entrypoint pid=364638) lora: (skyrl_entrypoint pid=364638) rank: 0 (skyrl_entrypoint pid=364638) alpha: 16 (skyrl_entrypoint pid=364638) dropout: 0 (skyrl_entrypoint pid=364638) lora_sync_path: /tmp/skyrl_lora_sync (skyrl_entrypoint pid=364638) target_modules: all-linear (skyrl_entrypoint pid=364638) exclude_modules: null (skyrl_entrypoint pid=364638) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=364638) optimizer_config: (skyrl_entrypoint pid=364638) optimizer: AdamW (skyrl_entrypoint pid=364638) lr: 8.0e-06 (skyrl_entrypoint pid=364638) adam_betas: (skyrl_entrypoint pid=364638) - 0.9 (skyrl_entrypoint pid=364638) - 0.999 (skyrl_entrypoint pid=364638) weight_decay: 0.0 (skyrl_entrypoint pid=364638) max_grad_norm: 0.9 (skyrl_entrypoint pid=364638) offload_after_step: true (skyrl_entrypoint pid=364638) num_warmup_steps: 0 (skyrl_entrypoint pid=364638) scheduler: constant_with_warmup (skyrl_entrypoint pid=364638) optimizer_kwargs: {} (skyrl_entrypoint pid=364638) fsdp_config: (skyrl_entrypoint pid=364638) cpu_offload: false (skyrl_entrypoint pid=364638) reshard_after_forward: true (skyrl_entrypoint pid=364638) fsdp_size: 4 (skyrl_entrypoint pid=364638) expert_model_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=364638) moe_token_dispatcher_type: alltoall (skyrl_entrypoint pid=364638) moe_router_replay: false (skyrl_entrypoint pid=364638) moe_grouped_gemm: false (skyrl_entrypoint pid=364638) ep_comm_backend: torch (skyrl_entrypoint pid=364638) deepep_num_sms: 20 (skyrl_entrypoint pid=364638) deepep_token_chunk_size: null (skyrl_entrypoint pid=364638) sequence_parallel_size: 1 (skyrl_entrypoint pid=364638) use_torch_compile: false (skyrl_entrypoint pid=364638) record_memory: false (skyrl_entrypoint pid=364638) megatron_config: (skyrl_entrypoint pid=364638) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=364638) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=364638) context_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_model_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_tensor_parallel_size: null (skyrl_entrypoint pid=364638) ddp_config: (skyrl_entrypoint pid=364638) grad_reduce_in_fp32: true (skyrl_entrypoint pid=364638) overlap_grad_reduce: false (skyrl_entrypoint pid=364638) overlap_param_gather: false (skyrl_entrypoint pid=364638) average_in_collective: true (skyrl_entrypoint pid=364638) model_config_kwargs: {} (skyrl_entrypoint pid=364638) torch_profiler_config: (skyrl_entrypoint pid=364638) enable: false (skyrl_entrypoint pid=364638) ranks: [] (skyrl_entrypoint pid=364638) save_path: null (skyrl_entrypoint pid=364638) optimizer_config_kwargs: (skyrl_entrypoint pid=364638) overlap_cpu_optimizer_d2h_h2d: false (skyrl_entrypoint pid=364638) use_precision_aware_optimizer: false (skyrl_entrypoint pid=364638) optimizer_cpu_offload: false (skyrl_entrypoint pid=364638) optimizer_offload_fraction: 0.0 (skyrl_entrypoint pid=364638) transformer_config_kwargs: (skyrl_entrypoint pid=364638) recompute_granularity: full (skyrl_entrypoint pid=364638) recompute_modules: (skyrl_entrypoint pid=364638) - core_attn (skyrl_entrypoint pid=364638) recompute_method: uniform (skyrl_entrypoint pid=364638) recompute_num_layers: 1 (skyrl_entrypoint pid=364638) empty_cuda_cache: true (skyrl_entrypoint pid=364638) ref: (skyrl_entrypoint pid=364638) model: (skyrl_entrypoint pid=364638) path: ${trainer.policy.model.path} (skyrl_entrypoint pid=364638) sequence_parallel_size: 1 (skyrl_entrypoint pid=364638) deepspeed_config: ${deepspeed_config.eval} (skyrl_entrypoint pid=364638) fsdp_config: (skyrl_entrypoint pid=364638) cpu_offload: false (skyrl_entrypoint pid=364638) reshard_after_forward: true (skyrl_entrypoint pid=364638) fsdp_size: 4 (skyrl_entrypoint pid=364638) expert_model_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=364638) moe_token_dispatcher_type: alltoall (skyrl_entrypoint pid=364638) moe_router_replay: false (skyrl_entrypoint pid=364638) moe_grouped_gemm: false (skyrl_entrypoint pid=364638) ep_comm_backend: torch (skyrl_entrypoint pid=364638) deepep_num_sms: 20 (skyrl_entrypoint pid=364638) deepep_token_chunk_size: null (skyrl_entrypoint pid=364638) megatron_config: (skyrl_entrypoint pid=364638) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=364638) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=364638) context_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_model_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=364638) model_config_kwargs: {} (skyrl_entrypoint pid=364638) transformer_config_kwargs: {} (skyrl_entrypoint pid=364638) critic: (skyrl_entrypoint pid=364638) model: (skyrl_entrypoint pid=364638) path: null (skyrl_entrypoint pid=364638) lora: (skyrl_entrypoint pid=364638) rank: 0 (skyrl_entrypoint pid=364638) alpha: 16 (skyrl_entrypoint pid=364638) dropout: 0 (skyrl_entrypoint pid=364638) target_modules: all-linear (skyrl_entrypoint pid=364638) exclude_modules: null (skyrl_entrypoint pid=364638) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=364638) optimizer_config: (skyrl_entrypoint pid=364638) optimizer: AdamW (skyrl_entrypoint pid=364638) lr: 5.0e-06 (skyrl_entrypoint pid=364638) adam_betas: (skyrl_entrypoint pid=364638) - 0.9 (skyrl_entrypoint pid=364638) - 0.999 (skyrl_entrypoint pid=364638) weight_decay: 0.01 (skyrl_entrypoint pid=364638) max_grad_norm: 1.0 (skyrl_entrypoint pid=364638) offload_after_step: true (skyrl_entrypoint pid=364638) num_warmup_steps: 0 (skyrl_entrypoint pid=364638) scheduler: constant_with_warmup (skyrl_entrypoint pid=364638) optimizer_kwargs: {} (skyrl_entrypoint pid=364638) fsdp_config: (skyrl_entrypoint pid=364638) cpu_offload: false (skyrl_entrypoint pid=364638) reshard_after_forward: true (skyrl_entrypoint pid=364638) fsdp_size: -1 (skyrl_entrypoint pid=364638) expert_model_parallel_size: 1 (skyrl_entrypoint pid=364638) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=364638) moe_token_dispatcher_type: alltoall (skyrl_entrypoint pid=364638) moe_router_replay: false (skyrl_entrypoint pid=364638) moe_grouped_gemm: false (skyrl_entrypoint pid=364638) ep_comm_backend: torch (skyrl_entrypoint pid=364638) deepep_num_sms: 20 (skyrl_entrypoint pid=364638) deepep_token_chunk_size: null (skyrl_entrypoint pid=364638) sequence_parallel_size: 1 (skyrl_entrypoint pid=364638) algorithm: (skyrl_entrypoint pid=364638) advantage_estimator: rloo_n (skyrl_entrypoint pid=364638) kl_ctrl: (skyrl_entrypoint pid=364638) type: fixed (skyrl_entrypoint pid=364638) kl_target: 0.1 (skyrl_entrypoint pid=364638) horizon: 10000 (skyrl_entrypoint pid=364638) kl_estimator_type: k3 (skyrl_entrypoint pid=364638) use_kl_estimator_k3: false (skyrl_entrypoint pid=364638) use_abs_kl: false (skyrl_entrypoint pid=364638) use_kl_in_reward: false (skyrl_entrypoint pid=364638) use_kl_loss: false (skyrl_entrypoint pid=364638) kl_loss_coef: 0.0 (skyrl_entrypoint pid=364638) use_entropy_loss: false (skyrl_entrypoint pid=364638) entropy_loss_coef: 0.01 (skyrl_entrypoint pid=364638) advantage_batch_normalize: false (skyrl_entrypoint pid=364638) value_head_prefix: value_head (skyrl_entrypoint pid=364638) policy_loss_type: regular (skyrl_entrypoint pid=364638) loss_reduction: seq_mean_token_sum_norm_global (skyrl_entrypoint pid=364638) global_loss_denom: null (skyrl_entrypoint pid=364638) grpo_norm_by_std: true (skyrl_entrypoint pid=364638) rloo_n_min_group_size: 4 (skyrl_entrypoint pid=364638) rloo_n_filter_zero_reward_groups: true (skyrl_entrypoint pid=364638) lambd: 1.0 (skyrl_entrypoint pid=364638) gamma: 1.0 (skyrl_entrypoint pid=364638) eps_clip_low: 0.2 (skyrl_entrypoint pid=364638) eps_clip_high: 0.05 (skyrl_entrypoint pid=364638) clip_ratio_c: 3.0 (skyrl_entrypoint pid=364638) tis_imp_ratio_cap: 2.0 (skyrl_entrypoint pid=364638) use_tis: true (skyrl_entrypoint pid=364638) sapo: (skyrl_entrypoint pid=364638) tau_pos: 1.0 (skyrl_entrypoint pid=364638) tau_neg: 1.05 (skyrl_entrypoint pid=364638) value_clip: 0.2 (skyrl_entrypoint pid=364638) dynamic_sampling: (skyrl_entrypoint pid=364638) type: null (skyrl_entrypoint pid=364638) max_sample_batches: 30 (skyrl_entrypoint pid=364638) min_replace_ratio: 0.3 (skyrl_entrypoint pid=364638) clip_cov: (skyrl_entrypoint pid=364638) clip_ratio: 0.0002 (skyrl_entrypoint pid=364638) clip_cov_lb: 1.0 (skyrl_entrypoint pid=364638) clip_cov_ub: 5.0 (skyrl_entrypoint pid=364638) kl_cov: (skyrl_entrypoint pid=364638) kl_cov_frac: 0.2 (skyrl_entrypoint pid=364638) ppo_kl_coef: 1.0 (skyrl_entrypoint pid=364638) cispo: (skyrl_entrypoint pid=364638) cispo_eps_clip_low: 0 (skyrl_entrypoint pid=364638) cispo_eps_clip_high: 5 (skyrl_entrypoint pid=364638) z_clip: (skyrl_entrypoint pid=364638) enabled: false (skyrl_entrypoint pid=364638) alpha: 0.97 (skyrl_entrypoint pid=364638) z_thresh: 2.5 (skyrl_entrypoint pid=364638) warmup_steps: 25 (skyrl_entrypoint pid=364638) mode: zscore (skyrl_entrypoint pid=364638) clip_option: adaptive_scaling (skyrl_entrypoint pid=364638) clip_factor: 1.0 (skyrl_entrypoint pid=364638) skip_update_on_spike: false (skyrl_entrypoint pid=364638) stale_clip: (skyrl_entrypoint pid=364638) enabled: false (skyrl_entrypoint pid=364638) alpha: 0.3 (skyrl_entrypoint pid=364638) entropy_threshold: 0.15 (skyrl_entrypoint pid=364638) entropy_window: 10 (skyrl_entrypoint pid=364638) min_lr_scale: 0.1 (skyrl_entrypoint pid=364638) max_seq_len: 1004095 (skyrl_entrypoint pid=364638) fully_async: (skyrl_entrypoint pid=364638) max_staleness_steps: 16 (skyrl_entrypoint pid=364638) num_parallel_generation_workers: 338 (skyrl_entrypoint pid=364638) gradient_checkpointing: true (skyrl_entrypoint pid=364638) gradient_checkpointing_use_reentrant: false (skyrl_entrypoint pid=364638) seed: 42 (skyrl_entrypoint pid=364638) resume_mode: latest (skyrl_entrypoint pid=364638) resume_path: null (skyrl_entrypoint pid=364638) ckpt_path: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints (skyrl_entrypoint pid=364638) max_ckpts_to_keep: -1 (skyrl_entrypoint pid=364638) ckpt_interval: 2 (skyrl_entrypoint pid=364638) hf_save_interval: 5 (skyrl_entrypoint pid=364638) hf_upload_mode: latest (skyrl_entrypoint pid=364638) export_path: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports (skyrl_entrypoint pid=364638) bf16: true (skyrl_entrypoint pid=364638) epochs: 2 (skyrl_entrypoint pid=364638) max_steps: 80 (skyrl_entrypoint pid=364638) update_epochs_per_batch: 1 (skyrl_entrypoint pid=364638) train_batch_size: 64 (skyrl_entrypoint pid=364638) policy_mini_batch_size: 64 (skyrl_entrypoint pid=364638) critic_mini_batch_size: 256 (skyrl_entrypoint pid=364638) micro_train_batch_size_per_gpu: 1 (skyrl_entrypoint pid=364638) micro_forward_batch_size_per_gpu: 4 (skyrl_entrypoint pid=364638) update_ref_every_epoch: false (skyrl_entrypoint pid=364638) use_sample_packing: true (skyrl_entrypoint pid=364638) eval_batch_size: 64 (skyrl_entrypoint pid=364638) eval_before_train: false (skyrl_entrypoint pid=364638) eval_interval: 999999 (skyrl_entrypoint pid=364638) max_prompt_length: 999999 (skyrl_entrypoint pid=364638) flash_attn: true (skyrl_entrypoint pid=364638) disable_fast_tokenizer: false (skyrl_entrypoint pid=364638) target_modules: null (skyrl_entrypoint pid=364638) exclude_modules: null (skyrl_entrypoint pid=364638) project_name: OpenThoughts-Agent (skyrl_entrypoint pid=364638) run_name: explore-tis-untrunc (skyrl_entrypoint pid=364638) logger: console (skyrl_entrypoint pid=364638) tracker_commit_each_step: true (skyrl_entrypoint pid=364638) dump_data_batch: false (skyrl_entrypoint pid=364638) dump_eval_results: true (skyrl_entrypoint pid=364638) log_level: INFO (skyrl_entrypoint pid=364638) rope_scaling: null (skyrl_entrypoint pid=364638) rope_theta: null (skyrl_entrypoint pid=364638) step_wise_training: false (skyrl_entrypoint pid=364638) hf_hub_repo_id: laion/explore-tis-untrunc (skyrl_entrypoint pid=364638) hf_hub_private: false (skyrl_entrypoint pid=364638) hf_hub_revision: main (skyrl_entrypoint pid=364638) enable_db_registration: false (skyrl_entrypoint pid=364638) generator: (skyrl_entrypoint pid=364638) model_name: ${trainer.policy.model.path} (skyrl_entrypoint pid=364638) model_dtype: bfloat16 (skyrl_entrypoint pid=364638) timeout_multiplier: 1.0 (skyrl_entrypoint pid=364638) run_engines_locally: true (skyrl_entrypoint pid=364638) num_inference_engines: 48 (skyrl_entrypoint pid=364638) backend: vllm (skyrl_entrypoint pid=364638) weight_sync_backend: nccl (skyrl_entrypoint pid=364638) fuse_weights: false (skyrl_entrypoint pid=364638) weight_transfer_threshold_cuda_ipc_GB: 1.0 (skyrl_entrypoint pid=364638) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=364638) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=364638) inference_engine_expert_parallel_size: 1 (skyrl_entrypoint pid=364638) inference_engine_data_parallel_size: 1 (skyrl_entrypoint pid=364638) inference_engine_mp_backend: false (skyrl_entrypoint pid=364638) n_samples_per_prompt: 8 (skyrl_entrypoint pid=364638) async_engine: true (skyrl_entrypoint pid=364638) batched: false (skyrl_entrypoint pid=364638) max_input_length: ${trainer.max_prompt_length} (skyrl_entrypoint pid=364638) vllm_v1_disable_multiproc: true (skyrl_entrypoint pid=364638) enable_prefix_caching: true (skyrl_entrypoint pid=364638) enable_chunked_prefill: true (skyrl_entrypoint pid=364638) max_num_batched_tokens: 65536 (skyrl_entrypoint pid=364638) enforce_eager: true (skyrl_entrypoint pid=364638) fully_sharded_loras: false (skyrl_entrypoint pid=364638) enable_ray_prometheus_stats: false (skyrl_entrypoint pid=364638) vllm_stats_interval: 1 (skyrl_entrypoint pid=364638) gpu_memory_utilization: 0.75 (skyrl_entrypoint pid=364638) max_num_seqs: 24 (skyrl_entrypoint pid=364638) remote_inference_engine_urls: (skyrl_entrypoint pid=364638) - 127.0.0.1:8001 (skyrl_entrypoint pid=364638) enable_http_endpoint: true (skyrl_entrypoint pid=364638) http_endpoint_host: 127.0.0.1 (skyrl_entrypoint pid=364638) http_endpoint_port: 8000 (skyrl_entrypoint pid=364638) max_turns: 999999 (skyrl_entrypoint pid=364638) chat_template: (skyrl_entrypoint pid=364638) source: name (skyrl_entrypoint pid=364638) name_or_path: null (skyrl_entrypoint pid=364638) chat_template_kwargs: {} (skyrl_entrypoint pid=364638) engine_init_kwargs: (skyrl_entrypoint pid=364638) max_model_len: 32768 (skyrl_entrypoint pid=364638) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=364638) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 (skyrl_entrypoint pid=364638) override_existing_update_group: disable (skyrl_entrypoint pid=364638) sampling_params: (skyrl_entrypoint pid=364638) max_generate_length: 4096 (skyrl_entrypoint pid=364638) repetition_penalty: 1.0 (skyrl_entrypoint pid=364638) temperature: 1.0 (skyrl_entrypoint pid=364638) top_p: 1.0 (skyrl_entrypoint pid=364638) min_p: 0.0 (skyrl_entrypoint pid=364638) top_k: -1 (skyrl_entrypoint pid=364638) logprobs: 0 (skyrl_entrypoint pid=364638) stop: null (skyrl_entrypoint pid=364638) use_conversation_multi_turn: true (skyrl_entrypoint pid=364638) append_eos_token_after_stop_str_in_multi_turn: true (skyrl_entrypoint pid=364638) eval_sampling_params: (skyrl_entrypoint pid=364638) max_generate_length: ${generator.sampling_params.max_generate_length} (skyrl_entrypoint pid=364638) repetition_penalty: 1.0 (skyrl_entrypoint pid=364638) temperature: 0.0 (skyrl_entrypoint pid=364638) top_p: 1.0 (skyrl_entrypoint pid=364638) min_p: 0.0 (skyrl_entrypoint pid=364638) top_k: -1 (skyrl_entrypoint pid=364638) logprobs: null (skyrl_entrypoint pid=364638) stop: null (skyrl_entrypoint pid=364638) eval_n_samples_per_prompt: 8 (skyrl_entrypoint pid=364638) zero_reward_on_non_stop: false (skyrl_entrypoint pid=364638) apply_overlong_filtering: false (skyrl_entrypoint pid=364638) rope_scaling: ${trainer.rope_scaling} (skyrl_entrypoint pid=364638) rope_theta: ${trainer.rope_theta} (skyrl_entrypoint pid=364638) teacher: (skyrl_entrypoint pid=364638) model_path: null (skyrl_entrypoint pid=364638) top_k_logprobs: 256 (skyrl_entrypoint pid=364638) num_inference_engines: 1 (skyrl_entrypoint pid=364638) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=364638) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=364638) gpu_memory_utilization: 0.9 (skyrl_entrypoint pid=364638) enforce_eager: false (skyrl_entrypoint pid=364638) backend: vllm (skyrl_entrypoint pid=364638) engine_init_kwargs: {} (skyrl_entrypoint pid=364638) environment: (skyrl_entrypoint pid=364638) env_class: gsm8k (skyrl_entrypoint pid=364638) skyrl_gym: (skyrl_entrypoint pid=364638) max_env_workers: 32 (skyrl_entrypoint pid=364638) text2sql: (skyrl_entrypoint pid=364638) db_path: /home/ray/default/sql_data (skyrl_entrypoint pid=364638) llm_as_a_judge: (skyrl_entrypoint pid=364638) model: gpt-4o-mini (skyrl_entrypoint pid=364638) base_url: null (skyrl_entrypoint pid=364638) search: (skyrl_entrypoint pid=364638) log_requests: false (skyrl_entrypoint pid=364638) search_url: http://127.0.0.1:8000/retrieve (skyrl_entrypoint pid=364638) topk: 3 (skyrl_entrypoint pid=364638) timeout: 30 (skyrl_entrypoint pid=364638) rollout: (skyrl_entrypoint pid=364638) fanout: (skyrl_entrypoint pid=364638) enabled: true (skyrl_entrypoint pid=364638) num_coordinators: 4 (skyrl_entrypoint pid=364638) cpus_per_coordinator: 8 (skyrl_entrypoint pid=364638) deepspeed_config: (skyrl_entrypoint pid=364638) train: (skyrl_entrypoint pid=364638) zero_optimization: (skyrl_entrypoint pid=364638) stage: 3 (skyrl_entrypoint pid=364638) offload_param: (skyrl_entrypoint pid=364638) device: none (skyrl_entrypoint pid=364638) offload_optimizer: (skyrl_entrypoint pid=364638) device: none (skyrl_entrypoint pid=364638) pin_memory: true (skyrl_entrypoint pid=364638) sub_group_size: auto (skyrl_entrypoint pid=364638) reduce_bucket_size: auto (skyrl_entrypoint pid=364638) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=364638) stage3_prefetch_bucket_size: auto (skyrl_entrypoint pid=364638) stage3_max_live_parameters: auto (skyrl_entrypoint pid=364638) stage3_max_reuse_distance: auto (skyrl_entrypoint pid=364638) round_robin_gradients: true (skyrl_entrypoint pid=364638) zero_hpz_partition_size: 1 (skyrl_entrypoint pid=364638) zero_quantized_weights: false (skyrl_entrypoint pid=364638) zero_quantized_gradients: false (skyrl_entrypoint pid=364638) torch_autocast: (skyrl_entrypoint pid=364638) enabled: true (skyrl_entrypoint pid=364638) dtype: bfloat16 (skyrl_entrypoint pid=364638) disable_trace_cache: false (skyrl_entrypoint pid=364638) data_types: (skyrl_entrypoint pid=364638) grad_accum_dtype: fp32 (skyrl_entrypoint pid=364638) gradient_clipping: 1.0 (skyrl_entrypoint pid=364638) wall_clock_breakdown: false (skyrl_entrypoint pid=364638) prescale_gradient: false (skyrl_entrypoint pid=364638) eval: (skyrl_entrypoint pid=364638) zero_optimization: (skyrl_entrypoint pid=364638) stage: 3 (skyrl_entrypoint pid=364638) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=364638) offload_param: (skyrl_entrypoint pid=364638) device: cpu (skyrl_entrypoint pid=364638) pin_memory: true (skyrl_entrypoint pid=364638) torch_autocast: (skyrl_entrypoint pid=364638) enabled: true (skyrl_entrypoint pid=364638) dtype: bfloat16 (skyrl_entrypoint pid=364638) gradient_clipping: 1.0 (skyrl_entrypoint pid=364638) prescale_gradient: false (skyrl_entrypoint pid=364638) wall_clock_breakdown: false (skyrl_entrypoint pid=364638) terminal_bench_config: (skyrl_entrypoint pid=364638) trials_dir: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/trace_jobs (skyrl_entrypoint pid=364638) harbor: (skyrl_entrypoint pid=364638) name: terminus-2 (skyrl_entrypoint pid=364638) max_episodes: 999999 (skyrl_entrypoint pid=364638) enable_summarize: false (skyrl_entrypoint pid=364638) store_all_messages: true (skyrl_entrypoint pid=364638) trajectory_config: (skyrl_entrypoint pid=364638) raw_content: true (skyrl_entrypoint pid=364638) enable_episode_logging: false (skyrl_entrypoint pid=364638) record_terminal_session: false (skyrl_entrypoint pid=364638) enable_pane_logging: false (skyrl_entrypoint pid=364638) strict_json_parser: true (skyrl_entrypoint pid=364638) interleaved_thinking: true (skyrl_entrypoint pid=364638) extra_body: (skyrl_entrypoint pid=364638) chat_template_kwargs: (skyrl_entrypoint pid=364638) enable_thinking: true (skyrl_entrypoint pid=364638) override_timeout_sec: 900 (skyrl_entrypoint pid=364638) override_cpus: 1 (skyrl_entrypoint pid=364638) override_memory_mb: 2048 (skyrl_entrypoint pid=364638) override_storage_mb: 2048 (skyrl_entrypoint pid=364638) auto_snapshot: true (skyrl_entrypoint pid=364638) verifier_override_timeout_sec: 120 (skyrl_entrypoint pid=364638) max_retries: 3 (skyrl_entrypoint pid=364638) min_wait_sec: 60.0 (skyrl_entrypoint pid=364638) max_wait_sec: 600.0 (skyrl_entrypoint pid=364638) wait_multiplier: 2.0 (skyrl_entrypoint pid=364638) exclude_exceptions: (skyrl_entrypoint pid=364638) - VerifierTimeoutError (skyrl_entrypoint pid=364638) - VerifierRuntimeError (skyrl_entrypoint pid=364638) - RewardFileNotFoundError (skyrl_entrypoint pid=364638) - RewardFileEmptyError (skyrl_entrypoint pid=364638) - VerifierOutputParseError (skyrl_entrypoint pid=364638) n_concurrent_trials: 675 (skyrl_entrypoint pid=364638) log_level: INFO (skyrl_entrypoint pid=364638) enable_reward_shaping: false (skyrl_entrypoint pid=364638) collect_rollout_details: true (skyrl_entrypoint pid=364638) enable_error_classification: true (skyrl_entrypoint pid=364638) mask_exceptions: (skyrl_entrypoint pid=364638) - DaytonaError (skyrl_entrypoint pid=364638) - EnvironmentStartTimeoutError (skyrl_entrypoint pid=364638) - NetworkError (skyrl_entrypoint pid=364638) - ConnectionError (skyrl_entrypoint pid=364638) - RewardFileNotFoundError (skyrl_entrypoint pid=364638) - RewardFileEmptyError (skyrl_entrypoint pid=364638) - AgentEnvironmentTimeoutError (skyrl_entrypoint pid=364638) - ContextLengthExceededError (skyrl_entrypoint pid=364638) default_error_treatment: zero (skyrl_entrypoint pid=364638) passthrough_exceptions: (skyrl_entrypoint pid=364638) - AgentTimeoutError (skyrl_entrypoint pid=364638) zero_exceptions: [] (skyrl_entrypoint pid=364638) model_info: (skyrl_entrypoint pid=364638) max_input_tokens: 32000 (skyrl_entrypoint pid=364638) max_output_tokens: 4096 (skyrl_entrypoint pid=364638) archiving: (skyrl_entrypoint pid=364638) enabled: false (skyrl_entrypoint pid=364638) trace_upload: (skyrl_entrypoint pid=364638) enabled: true (skyrl_entrypoint pid=364638) repo_org: DCAgent (skyrl_entrypoint pid=364638) episodes: last (skyrl_entrypoint pid=364638) dataset_type: SFT (skyrl_entrypoint pid=364638) cleanup: true (skyrl_entrypoint pid=364638)  (pid=348197) [2026-06-12 00:54:01,724 E 348197 363053] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 268x across cluster] (skyrl_entrypoint pid=364638) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (skyrl_entrypoint pid=364638) No module named 'vllm._version' (skyrl_entrypoint pid=364638) from .version import __version__, __version_tuple__ # isort:skip (skyrl_entrypoint pid=364638) W0612 00:54:09.087000 364638 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (raylet) 2026-06-12 00:54:12,109 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. (raylet, ip=10.128.25.23) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster] (raylet, ip=10.128.25.23) 2026-06-12 00:54:16,191 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 6x across cluster] (pid=478659, ip=10.128.25.176) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (pid=478659, ip=10.128.25.176) No module named 'vllm._version' (pid=478659, ip=10.128.25.176) from .version import __version__, __version_tuple__ # isort:skip (raylet, ip=10.128.25.161) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 78x across cluster] [2026-06-12 00:54:22,401 E 364224 364547] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.25.162) 2026-06-12 00:54:21,909 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 18x across cluster] (pid=316063, ip=10.128.25.23) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster] (pid=316063, ip=10.128.25.23) No module named 'vllm._version' [repeated 3x across cluster] (pid=316063, ip=10.128.25.23) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster] (RegistryActor pid=364573) [2026-06-12 00:54:23,544 E 364573 364613] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.25.163) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 90x across cluster] (raylet, ip=10.128.25.174) 2026-06-12 00:54:27,918 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 12x across cluster] (pid=249226, ip=10.128.25.24) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 20x across cluster] (pid=249226, ip=10.128.25.24) No module named 'vllm._version' [repeated 20x across cluster] (pid=249226, ip=10.128.25.24) from .version import __version__, __version_tuple__ # isort:skip [repeated 20x across cluster] (skyrl_entrypoint pid=364638) [2026-06-12 00:54:25,383 E 364638 364681] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster] (raylet, ip=10.128.25.165) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 56x across cluster] (raylet, ip=10.128.25.28) 2026-06-12 00:54:32,714 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 9x across cluster] (pid=443231, ip=10.128.25.174) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster] (pid=443231, ip=10.128.25.174) No module named 'vllm._version' [repeated 4x across cluster] (pid=443231, ip=10.128.25.174) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster] (skyrl_entrypoint pid=364638) [2026-06-12 00:54:36] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000 (skyrl_entrypoint pid=364638) [2026-06-12 00:54:36] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint... (skyrl_entrypoint pid=364638) [2026-06-12 00:54:37] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds) (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.317 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000 (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.317 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines. (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.319 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.320 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.321 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.322 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156 (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.322 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000 (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.322 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78 (skyrl_entrypoint pid=364638) 2026-06-12 00:54:37.322 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80 (raylet, ip=10.128.25.27) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 100x across cluster] (bundle_reservation_check_func pid=364729) [2026-06-12 00:54:42,131 E 364729 364769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.25.172) 2026-06-12 00:54:38,341 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 16x across cluster] (pid=326289, ip=10.128.25.27) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 20x across cluster] (pid=326289, ip=10.128.25.27) No module named 'vllm._version' [repeated 20x across cluster] (pid=326289, ip=10.128.25.27) from .version import __version__, __version_tuple__ # isort:skip [repeated 20x across cluster] (pid=441858, ip=10.128.25.172) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (raylet, ip=10.128.25.172) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 7x across cluster] (pid=316064, ip=10.128.25.23) [2026-06-12 00:54:48,004 E 316064 316166] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 7x across cluster] (pid=300698, ip=10.128.25.162) [2026-06-12 00:54:53,488 E 300698 300807] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (pid=443232, ip=10.128.25.174) [2026-06-12 00:54:59,702 E 443232 443308] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 12x across cluster] (pid=305554, ip=10.128.25.164) [2026-06-12 00:55:06,467 E 305554 305664] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:02.324 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING= (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:02.338 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:02.338 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity (pid=441858, ip=10.128.25.172) [2026-06-12 00:55:08,360 E 441858 441898] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:03.476 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0 (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:03.499 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:03.499 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:03.499 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:03.499 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=1.0, top_p=1.0, top_k=-1 (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) 2026-06-12 00:57:03.546 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.14s (attempt 1/5) to avoid port collisions (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) No module named 'vllm._version' (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) from .version import __version__, __version_tuple__ # isort:skip (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:03.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:03.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:03.394 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 2026-06-12 00:57:08 INFO [ipv4-debug] hostname=jpbo-029-44.jupiter.internal (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 2026-06-12 00:57:08 INFO [ipv4-debug] _global_node.node_ip_address=10.128.25.172 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 2026-06-12 00:57:08 INFO [ipv4-debug] get_node_ip_address()=10.128.25.172 (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:04.715 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:04.731 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:04.731 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:04.731 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 2026-06-12 00:57:04.731 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=1.0, top_p=1.0, top_k=-1 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25) 2026-06-12 00:57:04.700 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 1.80s (attempt 1/5) to avoid port collisions [repeated 47x across cluster] (skyrl_entrypoint pid=364638) 2026-06-12 00:57:08.981 | INFO  | skyrl_train.workers.worker:_initiate_actors:636 - Initializing process group for RayActorGroup (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [W612 00:57:09.772933846 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-44.jupiter.internal]:44913 (errno: 97 - Address family not supported by protocol). (raylet, ip=10.128.25.172) 2026-06-12 00:57:09,981 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. (pid=364947) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster] (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) No module named 'vllm._version' [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) _C._set_float32_matmul_precision(precision) (raylet) 2026-06-12 00:57:10,083 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. [repeated 6x across cluster] (pid=441955, ip=10.128.25.172) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) [W612 00:57:27.377459927 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-46-interconnect-1.jupiter.internal]:47029 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) [W612 00:57:27.379402447 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) [rank0]:[W612 00:57:27.384365575 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00 10.128.25.17 (routable head IP) for coordinator litellm base_url connectivity (skyrl_entrypoint pid=364638) 2026-06-12 01:01:38.239 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:405 - [RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8 (skyrl_entrypoint pid=364638) 2026-06-12 01:01:38.264 | INFO  | examples.terminal_bench.rollout_coordinator:startup:434 - [RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD) (raylet, ip=10.128.25.161) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (raylet, ip=10.128.25.161) 2026-06-12 01:01:39,065 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.319 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) [rank2]:[W612 01:01:35.573353744 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164) (EngineCore_DP0 pid=305964) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164) 2026-06-12 01:01:39.969 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 39x across cluster] (raylet, ip=10.128.25.161) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster] (skyrl_entrypoint pid=364638) 2026-06-12 01:01:44.531 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 1/4 started (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.526 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.526 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.526 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.527 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.25.17:8000) (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.530 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.530 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=350242, ip=10.128.25.161) 2026-06-12 01:01:44.530 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete (raylet, ip=10.128.25.176) 2026-06-12 01:01:47,209 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.275 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (RolloutCoordinator pid=350242, ip=10.128.25.161) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (raylet, ip=10.128.25.176) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster] (skyrl_entrypoint pid=364638) 2026-06-12 01:01:51.491 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 2/4 started (RolloutCoordinator pid=479438, ip=10.128.25.176) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.25.17:8000) (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.490 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.490 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=479438, ip=10.128.25.176) 2026-06-12 01:01:51.490 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete (raylet, ip=10.128.25.162) 2026-06-12 01:01:54,176 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.518 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (raylet, ip=10.128.25.162) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster] (RolloutCoordinator pid=301361, ip=10.128.25.162) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.25.17:8000) (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.743 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.743 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=301361, ip=10.128.25.162) 2026-06-12 01:01:58.743 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete (skyrl_entrypoint pid=364638) 2026-06-12 01:01:58.744 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 3/4 started (raylet, ip=10.128.25.25) 2026-06-12 01:02:01,691 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.745 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4) (raylet, ip=10.128.25.25) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster] (RolloutCoordinator pid=380389, ip=10.128.25.25) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.25.17:8000) (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.970 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.971 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168 (RolloutCoordinator pid=380389, ip=10.128.25.25) 2026-06-12 01:02:07.971 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.972 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 4/4 started (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.972 | INFO  | examples.terminal_bench.rollout_coordinator:startup:477 - [RolloutDispatcher] 4 coordinators started (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.972 | INFO  | skyrl_train.fully_async_trainer:train:485 - Generator startup complete (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.972 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Started: 'load_checkpoints' (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.975 | INFO  | skyrl_train.trainer:load_checkpoints:1735 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82 (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.975 | INFO  | skyrl_train.trainer:load_checkpoints:1741 - Resuming from global_step: 82 (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.981 | INFO  | skyrl_train.trainer:load_checkpoints:1757 - Successfully loaded trainer state (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.983 | INFO  | skyrl_train.trainer:load_checkpoints:1767 - Successfully loaded dataloader state (skyrl_entrypoint pid=364638) 2026-06-12 01:02:07.983 | INFO  | skyrl_train.trainer:load_checkpoints:1776 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy (RolloutCoordinator pid=350242, ip=10.128.25.161) [2026-06-12 01:02:09,083 E 350242 350282] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RolloutCoordinator pid=479438, ip=10.128.25.176) [2026-06-12 01:02:17,227 E 479438 479478] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RolloutCoordinator pid=301361, ip=10.128.25.162) [2026-06-12 01:02:24,194 E 301361 301401] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) warnings.warn( # warn only once (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.436 | INFO  | skyrl_train.trainer:load_checkpoints:1786 - Successfully loaded policy checkpoint (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.436 | INFO  | skyrl_train.trainer:load_checkpoints:1802 - Successfully loaded complete checkpoint state from global_step_82 (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.436 | INFO  | skyrl_train.fully_async_trainer:_train_loop:512 - Resumed training from global_step 82 (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.436 | WARNING  | skyrl_train.fully_async_trainer:_train_loop:517 - No data consumption state found in checkpoint — resume may re-train on already-consumed data (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.437 | WARNING  | skyrl_train.fully_async_trainer:_train_loop:537 - Data consumption count mismatch on resume: expected 256, got 0. This can happen after epoch boundary transitions or error recovery. (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.437 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Finished: 'load_checkpoints', time cost: 21.47s (skyrl_entrypoint pid=364638) 2026-06-12 01:02:29.437 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Started: 'init_weight_sync_state' (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) No module named 'vllm._version' (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) from .version import __version__, __version_tuple__ # isort:skip (RolloutCoordinator pid=380389, ip=10.128.25.25) [2026-06-12 01:02:31,710 E 380389 380429] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 2026-06-12 01:02:35.472 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.25.172, master_port=42815 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 2026-06-12 01:02:35.472 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.25.172 [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=364945) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) No module named 'vllm._version' [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165) (EngineCore_DP0 pid=304987) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) (EngineCore_DP0 pid=304984) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) [W612 01:05:00.280891184 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-37.jupiter.internal]:33847 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) [W612 01:05:00.281316200 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) [rank0]:[W612 01:05:00.286139823 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) WARNING 06-12 00:57:07 [arg_utils.py:1256] The global random seed is set to 68. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 00:57:07 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 00:57:07 [model.py:1549] Using max model len 32768 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 00:57:07 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 00:57:07 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 00:57:07 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) WARNING 06-12 00:57:07 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 00:57:07 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) WARNING 06-12 00:57:07 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) WARNING 06-12 00:57:07 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) INFO 06-12 00:57:26 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:19 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=78, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:27 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.174:47029 backend=nccl (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:27 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:29 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:44 [default_loader.py:293] Loading weights took 14.39 seconds (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:27 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:27 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.174:56675 backend=nccl [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:27 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:29 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:44 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.864247 seconds (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:48 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.25 seconds (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) WARNING 06-12 00:57:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:49 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) WARNING 06-12 00:57:49 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:49 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:44 [default_loader.py:293] Loading weights took 14.27 seconds [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) WARNING 06-12 00:57:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (skyrl_entrypoint pid=364638) [fd-monitor] [00:58:01] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=364638) [fd-monitor] [00:58:01] OK: RSS 1.64 GiB | node mem 137.6/858.0 GiB used (16.0%), avail 720.3 GiB (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:44 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.860758 seconds [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:48 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:48 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) (EngineCore_DP0 pid=443484) INFO 06-12 00:57:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) (EngineCore_DP0 pid=443488) INFO 06-12 00:57:48 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:48 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.22 seconds [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) WARNING 06-12 00:57:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:49 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) WARNING 06-12 00:57:49 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) (EngineCore_DP0 pid=443494) INFO 06-12 00:57:49 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) WARNING 06-12 00:57:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster] (skyrl_entrypoint pid=364638) [fd-monitor] [01:00:01] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=364638) [fd-monitor] [01:00:01] OK: RSS 1.64 GiB | node mem 137.7/858.0 GiB used (16.0%), avail 720.3 GiB (AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27) WARNING 06-12 01:00:55 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) INFO 06-12 01:00:55 [model.py:529] Resolved architecture: Qwen3ForCausalLM (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) INFO 06-12 01:00:55 [model.py:1549] Using max model len 32768 (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) INFO 06-12 01:00:55 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) INFO 06-12 01:00:55 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) INFO 06-12 01:00:55 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) WARNING 06-12 01:00:55 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) INFO 06-12 01:00:55 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176) WARNING 06-12 01:00:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163) WARNING 06-12 01:00:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned (pid=364947) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) WARNING 06-12 01:00:55 [arg_utils.py:1256] The global random seed is set to 65. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) INFO 06-12 01:00:55 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) INFO 06-12 01:00:55 [model.py:1549] Using max model len 32768 [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) INFO 06-12 01:00:55 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) INFO 06-12 01:00:55 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) INFO 06-12 01:00:55 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) WARNING 06-12 01:00:55 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) INFO 06-12 01:00:55 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) WARNING 06-12 01:00:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) WARNING 06-12 01:00:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) (EngineCore_DP0 pid=301668) INFO 06-12 01:01:09 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=66, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (pid=441955, ip=10.128.25.172) ⚙️ Running in WANDB offline mode [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) (EngineCore_DP0 pid=349999) INFO 06-12 01:01:15 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25) (EngineCore_DP0 pid=380144) INFO 06-12 01:01:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=49, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) (EngineCore_DP0 pid=349999) INFO 06-12 01:01:15 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.161:54467 backend=nccl (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) (EngineCore_DP0 pid=349999) INFO 06-12 01:01:16 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) (EngineCore_DP0 pid=349999) INFO 06-12 01:01:16 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... (AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161) (EngineCore_DP0 pid=349991) INFO 06-12 01:01:17 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162) (EngineCore_DP0 pid=301108) INFO 06-12 01:01:19 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176) (EngineCore_DP0 pid=479197) INFO 06-12 01:01:32 [default_loader.py:293] Loading weights took 14.07 seconds (AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162) (EngineCore_DP0 pid=301108) INFO 06-12 01:01:19 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.162:35647 backend=nccl [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25) (EngineCore_DP0 pid=380144) INFO 06-12 01:01:19 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25) (EngineCore_DP0 pid=380144) INFO 06-12 01:01:19 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162) (EngineCore_DP0 pid=301118) INFO 06-12 01:01:20 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176) (EngineCore_DP0 pid=479197) INFO 06-12 01:01:32 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.462349 seconds (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO cudaDriverVersion 13020 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.172<0> (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Initialized NET plugin IB (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Assigned NET plugin IB to comm (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Using network IB (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO DMA-BUF is available on GPU device 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO ncclCommInitRankConfig comm 0x400dfa6b40c0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 901000 commId 0xd750aa8fd3370a86 - Init START (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO comm 0x400dfa6b40c0 rank 0 nRanks 8 nNodes 2 localRanks 4 localRank 0 MNNVL 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/08 : 0 1 2 3 7 6 5 4 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 01/08 : 0 4 5 6 7 3 2 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/08 : 0 3 1 5 4 7 6 2 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 03/08 : 0 3 2 6 4 7 5 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/08 : 0 1 2 3 7 6 5 4 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 05/08 : 0 4 5 6 7 3 2 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/08 : 0 3 1 5 4 7 6 2 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 07/08 : 0 3 2 6 4 7 5 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Trees [0] 1/4/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->3 [4] 1/-1/-1->0->4 [5] -1/-1/-1->0->3 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->3 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO P2P Chunksize set to 131072 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO [Proxy Service] Device 0 CPU core 49 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442377 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c000000 size 2097152 ipcDesc 0x400ed4008b70 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c200000 size 2097152 ipcDesc 0x400ed4009b30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c400000 size 2097152 ipcDesc 0x400ed400a940 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 03/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c600000 size 2097152 ipcDesc 0x400ed400b730 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c800000 size 2097152 ipcDesc 0x400ed400c500 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 07/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118ca00000 size 2097152 ipcDesc 0x400ed400d2d0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442383 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 51 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/0 : 4[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/0 : 4[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 01/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 05/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118cc00000 size 10485760 ipcDesc 0x400ed4010180 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118d600000 size 10485760 ipcDesc 0x400ed4010f50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118e000000 size 10485760 ipcDesc 0x400ed4010f50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118ea00000 size 10485760 ipcDesc 0x400ed4010f50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118f400000 size 10485760 ipcDesc 0x400ed4010f50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118fe00000 size 10485760 ipcDesc 0x400ed4010f50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401190800000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401191200000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401191c00000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401192600000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401193000000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401193a00000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194400000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194600000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194800000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194a00000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194c00000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194e00000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195000000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195200000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195400000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195600000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195800000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 07/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195a00000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195c00000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195e00000 size 2097152 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401196000000 size 10485760 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401196a00000 size 10485760 ipcDesc 0x400ed4015c30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442374 [0] (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442347 [0] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442376 [0] NCCL INFO Allocated (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365456 [0] NCCL INFO Connected all trees (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365456 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365456 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365456 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365456 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e412e5990 rank 5 nranks 8 cudaDev 0 nvmlDev 1 busId 1901000 commId 0xd750aa8fd3370a86 - Init COMPLETE (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365456 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 5 nranks 8 total 2.83 (kernels 0.32, alloc 1.84, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.63, rest 0.00) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442346 [0] NCCL INFO CC Off, workFifoBytes 1048576 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 0 sendbuff 0x401340000000 recvbuff 0x401340000000 count 622329856 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 00/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 01/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 02/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 03/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 04/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 05/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 06/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 07/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 08/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 09/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 10/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 11/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 12/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 13/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 14/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 15/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 16/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 17/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 18/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 19/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 20/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 21/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 22/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 23/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 24/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 25/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 26/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 27/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 28/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 29/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 30/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 31/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 32/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 33/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 34/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 35/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 36/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 37/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 38/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 39/40 : 0 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[0] [send] via NE (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442389 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[1] [send] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) e400000 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442391 [0] NCCL INFO Channel 01/0 : 0[3] -> 1[3] [se (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) shareable buffer 0x400f1fc00000 size 2097152 ipcDesc 0x400f6429ffb0 (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) (EngineCore_DP0 pid=301662) INFO 06-12 01:01:36 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) (EngineCore_DP0 pid=301662) INFO 06-12 01:01:36 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) (EngineCore_DP0 pid=301662) INFO 06-12 01:01:36 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x (FSDPPolicyWorkerBase pid=364947) jpbo-026 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33: (FSDPPolicyWorkerBase pid=364944) jpbo (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) (EngineCore_DP0 pid=479189) INFO 06-12 01:01:36 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) T/IB/0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO Connected binomial trees (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442393 [0] NCCL INFO CC Off, workFifoBytes 1048576 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44: (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) via NET/IB/1 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo- (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) nd] via NET/IB/3 (AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28) (EngineCore_DP0 pid=317118) INFO 06-12 01:01:36 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.74 seconds (FSDPPolicyWorkerBase pid=364947) jp (FSDPPolicyWorkerBase pid=364946) fc00000 (FSDPPolicyWorkerBase pid=364944) ed30 (FSDPPolicyWorkerBase pid=364945) 400ff000ed30 (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Imported shareable buffer device 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442407 [0] NCCL INFO ncclCommInitRankConfig comm 0x400dfaa9fd00 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 901000 commId 0x8d21f48cdf4c109b - Ini (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:44 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) ff800ed30 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442412 [0] NCCL INFO Alloc (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) 0 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:4419 (FSDPPolicyWorkerBase pid=364947) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400f40000000 count 155582464 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365517 [ (FSDPPolicyWorkerBase pid=364946) IPC (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013528c0000 count 155582464 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401365180000 count 155582464 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401377a40000 count 155582464 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401100800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101000000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101800000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401116a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401116c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401116e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4011c3a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4011c3c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4011c3e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401100800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101000000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101800000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401135400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401138400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount a sendbuff 0x400edfa00000 recvbuff 0x400edfa00000 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00800 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01000 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01800 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount b sendbuff 0x400edfa31400 recvbuff 0x400edfa31400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa31c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401135c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401103000000 recvbuff 0x401103000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103200000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103400000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103600000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount a sendbuff 0x401103800000 recvbuff 0x401103800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401103800000 recvbuff 0x401103800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount b sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401135c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x400edfa03e00 recvbuff 0x400edfa03e00 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa03e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa03e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa03ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x400edfa04200 recvbuff 0x400edfa04200 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa042c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount d sendbuff 0x40113b400000 recvbuff 0x40113b400000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x400edfa00000 recvbuff 0x400edfa00000 count 4096 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00800 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01000 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01800 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x400edfa31c00 recvbuff 0x400edfa31c00 count 4096 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401107800000 recvbuff 0x401107800000 count 16777216 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401107c00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401107e00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108000000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa042c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113bc00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113c400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113d600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113d800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113da00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113b600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113b800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113ba00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113c400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113d400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401347800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134a800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341000000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401140e00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401141000000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401141200000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341200000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341a00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401342200000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa326c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401342200000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343a00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401345200000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349a00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ca00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401352a00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344200000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344a00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401345200000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401346400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401346600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401346800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343c00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343e00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344000000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401348a00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349200000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349a00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa336c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349a00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ca00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401352a00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401355a00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401358a00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ba00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134c200000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ca00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134c400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134cc00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ec00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135ac00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135c400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135dc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fc00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351a00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401353c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa362c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135c400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135dc00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401365400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ac00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401358e00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401359000000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401359200000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa362c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa376c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135c400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135dc00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401365400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136b400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ac00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3bc00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135e400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135ec00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135e400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135ec00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa386c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401360c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401366c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ac00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361c00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362e00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363000000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363200000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363a00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361c00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401366c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3e400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363a00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368a00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368c00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369e00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136a600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ae40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ae80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3aec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b2c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136a600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136be00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136d600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401377e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401379600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ae00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136ce00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136d600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136e800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136ea00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136ec00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401370e00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371e00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b2c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c6c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401377e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365515 [0 (FSDPPolicyWorkerBase pid=364946) (nil) recvbuff 0x401379600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ae00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137de00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401380e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384e00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401385600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401385000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401385800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d6c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3dac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401379600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ae00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137c600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401389a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401389c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401389e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3dac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ee40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ee80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3eec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137b600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ce00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137e600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137be00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137c600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ce00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401381800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401381a00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] (FSDPPolicyWorkerBase pid=364946) NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401381c00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137b800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ba00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137bc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137c600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ce00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137d600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x400edfa3fe00 recvbuff 0x400edfa3fe00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fe40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fe80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x400edfa40200 recvbuff 0x400edfa40200 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa402c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x401372600000 recvbuff 0x401372600000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x40164f000000 recvbuff 0x40164f000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401650800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401652000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401653800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x401372600000 recvbuff 0x401372600000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount af sendbuff 0x400edfa40c00 recvbuff 0x400edfa40c00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edf (FSDPPolicyWorkerBase pid=364946) a44c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401652000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137fa00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137fc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401652000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa402c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa416c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401656800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401658000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401659800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x400edfa43400 recvbuff 0x400edfa43400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x400edfa45400 recvbuff 0x400edfa45400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa45c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa46400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa46c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x401372600000 recvbuff 0x401372600000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x401374600000 recvbuff 0x401374600000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374a00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374c00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount be sendbuff 0x401372800000 recvbuff 0x401372800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x401373200000 recvbuff 0x401373200000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373a00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374200000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374a00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa426c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401656800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401658000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401659800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:3 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) t COMPLETE (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Broadcast: opCount 8b sendbuff 0x4010910000 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x401605800000 recvbuff 0x401605800000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x400edfa42a00 recvbuff 0x400edfa42a00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opC (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) ated shareable buffer 0x32e600000 size 2097152 ipcDesc 0x400ff800ed30 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO Broadcast: opCount 90 sendbuff 0x401091000000 recvbuff 0x401091000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e255 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) 00 recvbuff 0x401091000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e115b17e0 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Broadcast: opCount 63 sendbuff 0x400e99e13e00 recvbuff 0x400e99e13e00 count 4096 datatyp (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) af260 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:4419 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount 63 sendbuff 0x400e99e13e00 recvbuff 0x400e99e13e00 count 4096 datatype (FSDPPolicyWorkerBase pid=364946) ount 83 sendbuff 0x400edfa43e00 recvbuff 0x400edfa43e00 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x400edfa43e00 recvbuff 0x400edfa43e00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x401655000000 recvbuff 0x401655000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x40160b000000 recvbuff 0x40160b000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x401655000000 recvbuff 0x401655000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount da sendbuff 0x401655000000 recvbuff 0x401655000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount db sendbuff 0x400edfa45c00 recvbuff 0x400edfa45c00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x400edfa48400 recvbuff 0x400edfa48400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x401611000000 recvbuff 0x401611000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401613000000 recvbuff 0x401613000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401611000000 recvbuff 0x401611000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026- (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401617000000 recvbuff 0x401617000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x401617800000 recvbuff 0x401617800000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x400edfa47600 recvbuff 0x400edfa47600 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x400edfa47a00 recvbuff 0x400edfa47a00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401617000000 recvbuff 0x401617000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f9 sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 5033164 (FSDPPolicyWorkerBase pid=364945) size 10485760 ptr 0x334400000 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 8 sendbuff 0x400e99e02e00 recvbuff 0x400e99e02e00 count 4096 datatype 9 op 0 (FSDPPolicyWorkerBase pid=364946) 8 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x4013e6000000 recvbuff 0x4013e6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x400edfa4ac00 recvbuff 0x400edfa4ac00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400edfa4cc00 recvbuff 0x400edfa4cc00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x40161b800000 recvbuff 0x40161b800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount 102 sendbuff 0x400edfa49e00 recvbuff 0x400edfa49e00 count 128 datatype 9 op 0 ro (FSDPPolicyWorkerBase pid=364944) ] NCCL INFO Allocated shareable buffer 0x401024600000 size 10485760 ipcDesc 0x400fd000ed30 (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount 8 sendbuff 0x400e99e02e00 recvbuff 0x400e99e02e00 count 4096 datatype 9 op 0 r (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Br (FSDPPolicyWorkerBase pid=364947) root 0 comm 0x400e412e5990 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:36 (FSDPPolicyWorkerBase pid=364944) oot 0 comm 0x400e412e4700 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364 (FSDPPolicyWorkerBase pid=364945) oadcast: opCount 10 sendbuff 0x400eb9e00e00 recvbuff 0x400eb9e00e00 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 1f sendbuff 0x400ed9000000 recvbuff 0x400ed (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400ed0000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e6 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 37 sendbuff 0x400e99e07e00 recvbuff 0x400e99e07e00 coun (FSDPPolicyWorkerBase pid=364945) jpb (FSDPPolicyWorkerBase pid=364947) t 4096 datatype 9 op 0 root 0 comm 0x400e415b09a0 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL I (FSDPPolicyWorkerBase pid=364946) ot 0 comm 0x400e06983940 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x4013ea800000 recvbuff 0x4013ea800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x4013eb000000 recvbuff 0x4013eb000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount 10b sendbuff 0x4013eb000000 recvbuff 0x4013eb000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream ( (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL IN (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount (FSDPPolicyWorkerBase pid=364947) NFO Broadcast: opCount 37 sendbuff 0x400ed0c00000 recvbuff 0x400ed0c00000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) rec (FSDPPolicyWorkerBase pid=364944) FO Broadcast: opCount 37 sendbuff 0x400ed0c00000 recvbuff 0x400ed0c00000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recv (FSDPPolicyWorkerBase pid=364945) 4b sendbuff 0x400eef400000 recvbuff 0x400eef400000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e0b000 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nra (FSDPPolicyWorkerBase pid=364947) vbuff 0x400e99e0a600 count 1024 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364944) buff 0x400e99e0a600 count 1024 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount 53 sendbuff 0x401071000000 recvbuff 0x401071000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:36 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 53 sendbuff 0x401071000000 recvbuff 0x401071000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] strea (FSDPPolicyWorkerBase pid=364944) (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: (FSDPPolicyWorkerBase pid=364947) m (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast (FSDPPolicyWorkerBase pid=364944) opCount 5c sendbuff 0x401071000000 recvbuff 0x401071000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recv (FSDPPolicyWorkerBase pid=364947) : opCount 5c sendbuff 0x401071000000 recvbuff 0x401071000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) rec (FSDPPolicyWorkerBase pid=364946) nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x400edfa4a200 recvbuff 0x400edfa4a200 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x400edfa4b600 recvbuff 0x400edfa4b600 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x4013f2000000 recvbuff 0x4013f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount af sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: (FSDPPolicyWorkerBase pid=364944) buff 0x40107c000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401036c00000 recvbuff 0x401036c00000 count 50331648 datatype 9 op 0 (FSDPPolicyWorkerBase pid=364945) 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x401074000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4 (FSDPPolicyWorkerBase pid=364947) vbuff 0x40107c000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-0 (FSDPPolicyWorkerBase pid=364945) ] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadc (FSDPPolicyWorkerBase pid=364946) opCount 0 sendbuff (nil) recvbuff 0x4013e0800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) re (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadca (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount c1 sendbuff 0x400eb9e15200 rec (FSDPPolicyWorkerBase pid=364947) ast: opCount 83 sendbuff 0x400e99e16200 recvbuff 0x400e99e16200 count 128 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400f80000000 c (FSDPPolicyWorkerBase pid=364944) st: opCount 83 sendbuff 0x400e99e16200 recvbuff 0x400e99e16200 count 128 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount d0 sendbuff 0x400eb9e16e00 recvbuff 0x400eb9e16e00 count 4096 datatype 9 o (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) e 9 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029 (FSDPPolicyWorkerBase pid=364947) ount 12582912 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026 (FSDPPolicyWorkerBase pid=364945) p 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029- (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcas (FSDPPolicyWorkerBase pid=364947) jpbo-02 (FSDPPolicyWorkerBase pid=364946) cvbuff 0x400edfa4dc00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013ff800000 count 12582912 datatype 9 op 0 (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x400eb9e1be00 recvbuf (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400fa2 (FSDPPolicyWorkerBase pid=364945) f 0x400eb9e1be00 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e1ee00 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa4f240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (n (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:36494 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:3649 (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL IN (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL I (FSDPPolicyWorkerBase pid=364944) FO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e25400 count 32 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) INFO Broadcast: opCount 146 sendbuff 0x4010b2000000 recvbuff 0x4010b2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] N (FSDPPolicyWorkerBase pid=364947) NFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e25400 count 32 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) CCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e27e00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCou (FSDPPolicyWorkerBase pid=364946) il) (FSDPPolicyWorkerBase pid=364946) jpbo-026-3 (FSDPPolicyWorkerBase pid=364945) nt 163 sendbuff 0x4010b2000000 recvbuff 0x4010b2000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 172 sendbuff 0x4010e0000000 recvbuff (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e415b1b60 [nranks=2] st (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e415b09a0 [nranks=2] s (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) t: opCount 26 sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010efa00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [ (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: (FSDPPolicyWorkerBase pid=364946) opCount 0 sendbuff (nil) recvbuff 0x400edfa542c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401367c00000 count 12582912 datatype 9 op 0 r (AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28) (EngineCore_DP0 pid=317128) WARNING 06-12 01:01:36 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28) (EngineCore_DP0 pid=317128) INFO 06-12 01:01:36 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28) (EngineCore_DP0 pid=317128) WARNING 06-12 01:01:36 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28) (EngineCore_DP0 pid=317128) INFO 06-12 01:01:36 [vllm.py:846] Cudagraph is disabled under eager mode (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ac00000 count 4194304 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nra (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) WARNING 06-12 01:01:37 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) r (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) ecvbuff 0x4015d2200000 count 4194304 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:35 [default_loader.py:293] Loading weights took 15.14 seconds [repeated 39x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013c5800000 count 1 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount e8 sendbuff 0x401626000000 recvbuf (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858 (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:36 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 16.330888 seconds [repeated 39x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:01:44] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:01:44] OK: RSS 0.78 GiB | node mem 372.6/858.0 GiB used (43.4%), avail 485.4 GiB (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO cudaDriverVersion 13020 [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365455 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.  [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365455 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.17<0> [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365455 [0] NCCL INFO Initialized NET plugin IB [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Assigned NET plugin IB to comm [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Using network IB [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO DMA-BUF is available on GPU device 0 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e616c3ee0 rank 3 nranks 4 cudaDev 0 nvmlDev 3 busId 3901000 commId 0xa71b72472f667c3e - Init START [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365455 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365455 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Setting affinity for GPU 3 to 216-287 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO comm 0x400e616c3ee0 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365511 [0] NCCL INFO Channel 23/24 : 0 3 2 1 [repeated 168x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2 [2] -1/-1/-1->3->2 [3] -1/-1/-1->3->2 [4] -1/-1/-1->3->1 [5] -1/-1/-1->3->1 [6] -1/-1/-1->3->1 [7] -1/-1/-1->3->1 [8] 2/-1/-1->3->0 [9] 2/-1/-1->3->0 [10] 2/-1/-1->3->0 [11] 2/-1/-1->3->0 [12] -1/-1/-1->3->2 [13] -1/-1/-1->3->2 [14] -1/-1/-1->3->2 [15] -1/-1/-1->3->2 [16] -1/-1/-1->3->1 [17] -1/-1/-1->3->1 [18] -1/-1/-1->3->1 [19] -1/-1/-1->3->1 [20] 2/-1/-1->3->0 [21] 2/-1/-1->3->0 [22] 2/-1/-1->3->0 [23] 2/-1/-1->3->0 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO P2P Chunksize set to 524288 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365455 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.  [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365511 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365516 [0] NCCL INFO [Proxy Service] Device 0 CPU core 274 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365519 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 276 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442410 [0] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC [repeated 586x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365515 [0] NCCL INFO Allocated shareable buffer 0x401028200000 size 10485760 ipcDesc 0x400fd000ed30 [repeated 1175x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365523 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 216 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365494 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[3] [receive] via NET/IB/3 [repeated 346x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365494 [0] NCCL INFO Channel 39/0 : 1[3] -> 0[3] [send] via NET/IB/3 [repeated 340x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365529 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x40102aa00000 [repeated 1187x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442408 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Connected all trees [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 16 p2p channels per peer [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442349 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e616c3ee0 rank 3 nranks 4 cudaDev 0 nvmlDev 3 busId 3901000 commId 0xa71b72472f667c3e - Init COMPLETE [repeated 22x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365512 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 3 nranks 4 total 0.17 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.14, rest 0.00) [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365511 [0] NCCL INFO CC Off, workFifoBytes 1048576 [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 186 sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) [repeated 5447x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442390 [0] NCCL INFO Channel 01/0 : 0[2] -> 1[2] [se (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:38 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:38 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:38 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 39x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:39 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 39x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365494 [0] NCCL INFO Connected binomial trees [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) nd] via NET/IB/2 (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) (EngineCore_DP0 pid=305972) INFO 06-12 01:01:39 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.08 seconds [repeated 39x across cluster] (FSDPPolicyWorkerBase pid=364945) jp (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010d8200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=4] stream (nil) [repeated 2329x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [ (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02ac0 count 32 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) [repeated 1758x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount a sendbuff 0x401103000000 recvbuff 0x401103000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount b sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401107800000 recvbuff 0x401107800000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=364947) (nil) recvbuff 0x400eae800000 count 12582912 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x4013a0000000 recvbuff 0x4013a0000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x4013a0000000 recvbuff 0x4013a0000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x4013a0000000 recvbuff 0x4013a0000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x4013a6000000 recvbuff 0x4013a6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 22x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x4013a6000000 recvbuff 0x4013a6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount af sendbuff 0x4013a6000000 recvbuff 0x4013a6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013af000 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount be sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 15e sendbuff 0x401 [repeated 4x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 128 sendbuff 0x400eb9e20e00 recvbuff 0x400eb9e20e00 count 4096 datatype 9 op 0 root 0 comm 0x400e615a [repeated 8x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) recvbuff 0x40134c400000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) nks=4] stream (nil) [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 95 sendbuff 0x400eb9e10200 recvbuff 0x400eb9e10200 count 128 datatype (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x4013b5c00000 recvbuff 0x4013b5c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount da sendbuff 0x401620000000 recvbuff 0x401620000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount db sendbuff 0x4013c6800000 recvbuff 0x4013c6800000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x4013c6800000 recvbuff 0x4013c6800000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x401620400000 recvbuff 0x401620400000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401626000000 recvbuff 0x401626000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401626000000 recvbuff 0x401626000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401626000000 recvbuff 0x401626000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x401628800000 recvbuff 0x401628800000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x401630800000 recvbuff 0x401630800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x401630800000 recvbuff 0x401630800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401630800000 recvbuff 0x401630800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x401630800000 recvbuff 0x401630800000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 27x across cluster] (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount e9 sendbuff 0x4010a4000000 recvbuff 0x4010a4000000 count 4194 [repeated 6x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 2582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) [repeated 20x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x4015d1a00000 recvbuff 0x4015d1a00000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) [repeated 12x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x400e9fa4b400 recvbuff 0x400e9fa4b400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) [repeated 10x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa4b400 recvbuff 0x400e9fa4b400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) [repeated 9x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4015d1a00000 recvbuff 0x4015d1a00000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) [repeated 9x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x4015d1a00000 recvbuff 0x4015d1a00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) [repeated 9x across cluster] (FSDPPolicyWorkerBase pid=364946) root 1 comm 0x400e06a9e080 [nranks=4] stream (nil) [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) oot 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2a400 count 32 datatype 9 op 0 root 0 comm 0x400e41 [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount a8 (FSDPPolicyWorkerBase pid=364944) sendbuff 0x4010a0000000 recvbuff 0x4010a0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) vbuff 0x400eb9e15200 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401034e00000 recvbuff 0x401034e00000 count 50331648 datatype 9 op 0 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2c (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) f 0x401626000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL  [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164) (EngineCore_DP0 pid=305964) WARNING 06-12 01:01:39 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164) (EngineCore_DP0 pid=305964) INFO 06-12 01:01:39 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164) (EngineCore_DP0 pid=305964) WARNING 06-12 01:01:39 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164) (EngineCore_DP0 pid=305964) INFO 06-12 01:01:39 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164) WARNING 06-12 01:01:39 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 39x across cluster] (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:01:51] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:01:51] OK: RSS 0.78 GiB | node mem 359.9/858.0 GiB used (41.9%), avail 498.1 GiB (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:01:58] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:01:58] OK: RSS 0.78 GiB | node mem 371.0/858.0 GiB used (43.2%), avail 486.9 GiB (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] Started monitoring (every 120s) (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:02:07] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) [repeated 2x across cluster] (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:02:07] OK: RSS 0.78 GiB | node mem 381.1/858.0 GiB used (44.4%), avail 476.9 GiB [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy/model_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy/extra_state_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy/optim_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Successfully loaded model state dict (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Successfully loaded optimizer state (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Successfully loaded scheduler state (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy (FSDPPolicyWorkerBase pid=364945) e00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 181 sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 181 sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2ae00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 182 sendbuff 0x4010ec000000 recvbuff 0x4010ec000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 182 sendbuff 0x4010ec000000 recvbuff 0x4010ec000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010e5000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 183 sendbuff 0x4010e5800000 recvbuff 0x4010e5800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 183 sendbuff 0x4010e5800000 recvbuff 0x4010e5800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010ec000000 count 1048576 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 184 sendbuff 0x4010ec200000 recvbuff 0x4010ec200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 184 sendbuff 0x4010ec200000 recvbuff 0x4010ec200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010e5800000 count 1048576 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 185 sendbuff 0x4010eca00000 recvbuff 0x4010eca00000 count 16777216 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 185 sendbuff 0x4010eca00000 recvbuff 0x4010eca00000 count 16777216 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010ec200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 186 sendbuff 0x400eb9e2b600 recvbuff 0x400eb9e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 186 sendbuff 0x400eb9e2b600 recvbuff 0x400eb9e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2b800 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 187 sendbuff 0x400eb9e2ba00 recvbuff 0x400eb9e2ba00 count 128 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 187 sendbuff 0x400eb9e2ba00 recvbuff 0x400eb9e2ba00 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2b600 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 188 sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 188 sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010eca00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 189 sendbuff 0x4010f8000000 recvbuff 0x4010f8000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 189 sendbuff 0x4010f8000000 recvbuff 0x4010f8000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010ee200000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010efa00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18b sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18b sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2ba00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18c sendbuff 0x400eb9e2f600 recvbuff 0x400eb9e2f600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18c sendbuff 0x400eb9e2f600 recvbuff 0x400eb9e2f600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2c200 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18d sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18d sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2f600 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18e sendbuff 0x401100000000 recvbuff 0x401100000000 count 622329856 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18e sendbuff 0x401100000000 recvbuff 0x401100000000 count 622329856 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x40114a400000 count 155582464 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:03:44] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:03:44] OK: RSS 0.89 GiB | node mem 372.8/858.0 GiB used (43.4%), avail 485.2 GiB (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:03:51] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:03:51] OK: RSS 0.89 GiB | node mem 360.2/858.0 GiB used (42.0%), avail 497.8 GiB (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:03:58] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:03:58] OK: RSS 0.89 GiB | node mem 371.2/858.0 GiB used (43.3%), avail 486.8 GiB (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:04:07] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) [repeated 2x across cluster] (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:04:07] OK: RSS 0.89 GiB | node mem 381.3/858.0 GiB used (44.4%), avail 476.7 GiB [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:04:59 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:00 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.165:33847 backend=nccl (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:00 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:19 [default_loader.py:293] Loading weights took 15.15 seconds (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:04:59 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:00 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.165:34269 backend=nccl [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:00 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:20 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.668826 seconds (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:23 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:23 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:23 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) (EngineCore_DP0 pid=304984) INFO 06-12 01:05:24 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:24 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.09 seconds (AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165) (EngineCore_DP0 pid=304987) WARNING 06-12 01:05:24 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:19 [default_loader.py:293] Loading weights took 15.15 seconds [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:24 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) WARNING 06-12 01:05:24 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) (EngineCore_DP0 pid=304985) INFO 06-12 01:05:24 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) WARNING 06-12 01:05:25 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) :441948 [0] NCCL INFO Broadcast: opCount 187 sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02a40 count 32 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02a80 count 32 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02ac0 count 32 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 188 sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401641800000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401643000000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401644800000 count 12582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 189 sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401641800000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401643000000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401644800000 count 12582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401641800000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401643000000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401644800000 count 12582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18b sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5ac00 count 1024 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5b400 count 1024 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5bc00 count 1024 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18c sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5ac00 count 1024 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5b400 count 1024 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5bc00 count 1024 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18d sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5ac00 count 1024 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5b400 count 1024 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5bc00 count 1024 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x401660000000 recvbuff 0x401660000000 count 622329856 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18e sendbuff 0x401660000000 recvbuff 0x401660000000 count 622329856 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4016728c0000 count 155582464 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401685180000 count 155582464 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401697a40000 count 155582464 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:301668 [0] NCCL INFO cudaDriverVersion 13020 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:301668 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.163<0> (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Initialized NET plugin IB (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Assigned NET plugin IB to comm (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Using network IB (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO DMA-BUF is available on GPU device 0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab0c759090 rank 25 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init START (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO comm 0xaaab0c759090 rank 25 nRanks 49 nNodes 13 localRanks 4 localRank 0 MNNVL 0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Trees [0] 38/-1/-1->25->21 [1] -1/-1/-1->25->40 [2] 38/-1/-1->25->39 [3] 39/-1/-1->25->40 [4] 38/21/26->25->17 [5] -1/-1/-1->25->40 [6] 38/-1/-1->25->39 [7] 39/-1/-1->25->40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO P2P Chunksize set to 131072 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO [Proxy Service] Device 0 CPU core 24 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302095 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 25 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302099 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 26 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 00/0 : 25[0] -> 26[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 25[0] -> 26[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 01/0 : 21[0] -> 25[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 05/0 : 21[0] -> 25[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 01/0 : 25[0] -> 38[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790800000 size 2097152 ipcDesc 0x40096400ae40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 05/0 : 25[0] -> 38[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790a00000 size 2097152 ipcDesc 0x40096400bc60 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 02/0 : 25[0] -> 40[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790c00000 size 2097152 ipcDesc 0x40096400ca30 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 03/0 : 25[0] -> 40[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790e00000 size 2097152 ipcDesc 0x40096400d800 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 06/0 : 25[0] -> 40[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791000000 size 2097152 ipcDesc 0x40096400e5d0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 07/0 : 25[0] -> 40[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791200000 size 2097152 ipcDesc 0x40096400f3a0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791400000 size 10485760 ipcDesc 0x400964010170 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791e00000 size 10485760 ipcDesc 0x400964010f40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401792800000 size 10485760 ipcDesc 0x400964010f40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401793200000 size 10485760 ipcDesc 0x400964010f40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401793c00000 size 10485760 ipcDesc 0x400964010f40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401794600000 size 10485760 ipcDesc 0x400964010f40 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401795000000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401795a00000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401796400000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401796e00000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401797800000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401798200000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401798c00000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401798e00000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799000000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799200000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799400000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799600000 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 00/0 : 21[0] -> 25[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 21[0] -> 25[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 17[0] -> 25[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 00/0 : 25[0] -> 38[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799800000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 02/0 : 25[0] -> 38[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799a00000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 25[0] -> 38[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799c00000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 06/0 : 25[0] -> 38[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799e00000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 02/0 : 25[0] -> 39[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a000000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 03/0 : 25[0] -> 39[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a200000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 06/0 : 25[0] -> 39[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a400000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 07/0 : 25[0] -> 39[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a600000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 01/0 : 25[0] -> 40[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a800000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302065 [0] NCCL INFO Channel 05/0 : 25[0] -> 40[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179aa00000 size 2097152 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179ac00000 size 10485760 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179b600000 size 10485760 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179c000000 size 10485760 ipcDesc 0x400964016140 (AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163) jpbo-0 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:20 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.667732 seconds [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:23 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:23 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:23 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163) jpb (AsyncVLLMInferenceEngine pid=315936, ip=10.128.25.23) jpbo-026-39:316481:316921 [0] NCCL INFO Allocated shareable buffer 0x4017bd000000 (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) jpbo-029-33:349999:350525 [0] NCCL INFO Allocated shareable buffer 0x4017bc600000 size 1 (AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23) jpbo-026-39:316471:316887 [0] NCCL INFO Channel 00/0 : 4[2] -> (AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161) jpbo-029 (AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23) jpbo-026-39:316483:316889 [0] NCCL INFO Channel 00/0 (AsyncVLLMInferenceEngine pid=349500, ip=10.128.25.161) jpbo-029-33:349997:35 (AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161) jp (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) jpbo-026-39:316475:3 (AsyncVLLMInferenceEngine pid=300569, ip=10.128.25.162) jpbo- (AsyncVLLMInferenceEngine pid=300697, ip=10.128.25.162) jpbo-0 (AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28) jpbo-026-44:317118:317547 [0] NCCL INFO Allocat (AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28) jpbo-02 (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) jpbo-0 (AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165) jpbo-0 (AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176) jpbo-029-48:479197:4 (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) jpbo-029-48:479189:479707 [0] NCCL INFO Allocated shareable buffer 0x4017bfc00000 size 2097152 ipcDes (AsyncVLLMInferenceEngine pid=478791, ip=10.128.25.176) jpbo-029-48:47919 (AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176) jpbo-029-48:479185:479704 [0] NCCL INFO Allocated shareable buffer 0x40179e200000 s (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:327159 [0] NCCL INFO Imported shareable buffer de (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:250073 [0] NCCL INFO Allocated (AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24) jpbo-02 (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo-026-41:380145:380620 [0] NCCL INFO Connected all trees (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo-026-41:380145:380620 [0] NCCL INFO threadThresholds 8/8/64 | 392/8/64 | 512 | 512 (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo-026-41:380145:380620 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 1 p2p channels per peer (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo-026-41:380145:380620 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo-026-41:380145:380620 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab14413a40 rank 5 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init COMPLETE (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo-026-41:380145:380620 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 5 nranks 49 total 3.38 (kernels 0.32, alloc 2.10, bootstrap 0.28, allgathers 0.01, topo 0.03, graphs 0.01, connections 0.63, rest 0.01) (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) 33f250 (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) 095833f6f0 (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) 0485760 ipcDesc 0x40097033fc50 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=315936, ip=10.128.25.23) size 10485760 ipcDesc 0x40097833bd20 (AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23) 3[1] via P2P/IPC (AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23) : 3[1] -> 2[0] via P2P/IPC (AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162) d20 (AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162) 433fbc0 (AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28) ed shareable buffer 0x40179c600000 size 10485760 ipcDesc 0x40095c33fd50 (AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28) 5033d590 (AsyncVLLMInferenceEngine pid=316639, ip=10.128.25.28) 96833fda0 (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) 33fd70 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) 833fc30 (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) c 0x4009781d4530 (AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176) ize 2097152 ipcDesc 0x400970015eb0 (AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27) c60 (AsyncVLLMInferenceEngine pid=326287, ip=10.128.25.27) 433eca0 (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) vice 0 size 2097152 ptr 0x4017be200000 (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) shareable buffer 0x40179c600000 size 10485760 ipcDesc 0x40096833fcd0 (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) 3fc90 (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 95433fe50 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount ff sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401820000000 recvbuff 0x401660000000 count 155582464 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 00/08 : 0 16 15 14 1 2 3 4 13 8 7 6 5 9 10 11 12 20 19 18 17 21 22 23 24 40 39 38 25 26 27 28 37 32 31 30 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 01/08 : 0 1 14 15 16 13 4 3 2 5 6 7 8 12 11 10 9 17 18 19 20 24 23 22 21 25 38 39 40 37 28 27 26 29 30 31 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 02/08 : 0 14 1 16 15 4 2 13 3 6 5 8 7 11 9 12 10 18 17 20 19 23 21 24 22 38 25 40 39 28 26 37 27 30 29 32 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 03/08 : 0 15 1 16 14 3 2 13 4 7 5 8 6 10 9 12 11 19 17 20 18 22 21 24 23 39 25 40 38 27 26 37 28 31 29 32 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 04/08 : 0 16 15 14 1 2 3 4 13 8 7 6 5 9 10 11 12 20 19 18 17 21 22 23 24 40 39 38 25 26 27 28 37 32 31 30 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 05/08 : 0 1 14 15 16 13 4 3 2 5 6 7 8 12 11 10 9 17 18 19 20 24 23 22 21 25 38 39 40 37 28 27 26 29 30 31 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 06/08 : 0 14 1 16 15 4 2 13 3 6 5 8 7 11 9 12 10 18 17 20 19 23 21 24 22 38 25 40 39 28 26 37 27 30 29 32 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 07/08 : 0 15 1 16 14 3 2 13 4 7 5 8 6 10 9 12 11 19 17 20 18 22 21 24 23 39 25 40 38 27 26 37 28 31 29 32 ... (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO CC Off, workFifoBytes 1048576 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 0 sendbuff 0x401660000000 recvbuff 0x401660000000 count 622329856 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 100 sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401635c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 1 sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 101 sendbuff 0x400e9 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.08 seconds [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opC (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) INFO Broadcast: opCount fa sendbuff 0x400e99e2b600 recvbuff 0x400e99e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2b800 count 32 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2b600 count 32 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010f8a00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010fa200000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010fba00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2ba00 count 1024 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2c200 count 1024 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2f600 count 1024 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x40116a400000 count 155582464 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4012f3e00000 recvbuff 0x401760000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] strea (FSDPPolicyWorkerBase pid=364947) tream (nil) (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: op (FSDPPolicyWorkerBase pid=364944) ream (nil) (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) fa88c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount a sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount b sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401660000000 recvbuff 0x401660000000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO Al (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) nranks=4] stream (nil) (FSDPPolicyWorkerBase pid=364946) ount 108 sendbuff 0x400edfa88c00 recvbuff 0x400edfa88c00 count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendb (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) lGather: opCount 0 sendbuff 0x400e9fa5b600 recvbuff 0x400e9fa88e00 count 32 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) m 0x400e113f9600 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) 0x400e093f6c80 (FSDPPolicyWorkerBase pid=364947) Count 0 sendbuff 0x4012d3a00000 recvbuff 0x4017aa400000 count 1048576 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160 (FSDPPolicyWorkerBase pid=364947) jpbo-026 (FSDPPolicyWorkerBase pid=364944) ount 0 sendbuff 0x4012d3a00000 recvbuff 0x4017aa400000 count 1048576 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020 (FSDPPolicyWorkerBase pid=364944) jpbo-026- (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) jpbo-029-46:443488:443488 [0] NCCL INFO Broadcast: opCount 2a sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab28ec7140 [nran (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) jpbo-029-46:443505:443505 [0] NCCL INFO Broadcast: opCount 1c sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab032e5280 [nranks=49] s (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) WARNING 06-12 01:05:24 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401860000000 recvbuff 0x4016a0000000 count 12582912 datatype 9 o (FSDPPolicyWorkerBase pid=364945) uff 0x400eb9e2ea00 recvbuff 0x400eb9e88e00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) c8310 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 2c sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400e99e32c00 recvbuff 0x400e99 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 13a sendbuff 0x400e9fa8 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) WARNING 06-12 01:05:24 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4010f4a0 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e412e5990 [ (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [n (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) e88c00 count 32 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream 0x400e113f9600 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCoun (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) WARNING 06-12 01:05:25 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 3a sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) [repeated 1717x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4016d7a40000 count 155582464 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil) [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) [repeated 4x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x4010e6000000 recvbuff 0x4010e6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x401100000000 recvbuff 0x401100000000 count 622329856 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) [repeated 20x across cluster] (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) [repeated 240x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount (FSDPPolicyWorkerBase pid=364946) p 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 148 sendbuf (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 8c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) 0000 recvbuff 0x401718c00000 count 1048576 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:44 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4012e9400000 recvbuff 0x4017aa400000 count 4194304 datatype (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) jpbo-029-37:304985:304985 [0] NCCL INFO Broadcast: opCount 55 sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root (AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165) jpbo-029-37:304987:304987 [0] NCCL INFO Broadcast: opCount 37 sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) : opCount 0 sendbuff 0x40184e400000 recvbuff 0x401660000000 count 12582912 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) t fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:44 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442 (FSDPPolicyWorkerBase pid=364947) 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 (FSDPPolicyWorkerBase pid=364946) f 0x400edfa89000 recvbuff 0x400edfa89000 count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364946) jpbo-026-33: (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 r (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 65 sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 dat (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:3649 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) atype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 173 sendbuff 0x400e9 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400faa000000 recvbuff 0x401760000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: o (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400edfa67a00 recvbuff 0x400edfa88e00 count 32 datatype 9 op 0 root 0 (FSDPPolicyWorkerBase pid=364944) ecvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e3ae00 recvbu (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) jpbo-029-46:443488:443488 [0] NCCL INFO Broadcast: opCount 80 sendbuff 0x401720000000 recvbuff 0x401720000000 count 503 (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) jpbo-029-46:443505:443505 [0] NCCL INFO Broadcast: opCount 52 sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 da (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) ] stream 0x400e113f9600 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 o (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) stream 0x400e093f6c80 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op (FSDPPolicyWorkerBase pid=364947) pCount 0 sendbuff 0x400e99e3ae00 recvbuff 0x400e99e8b000 count 1024 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160 (FSDPPolicyWorkerBase pid=364947) jpbo-02 (FSDPPolicyWorkerBase pid=364946) comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0 (FSDPPolicyWorkerBase pid=364944) jpbo-026 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) INFO AllGather: opCount 0 sendbuff 0x40186e800000 recvbuff 0x401660000000 count 12582912 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401870800000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] str (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) ff 0x400eb9e8b000 count 1024 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e3e000 recvbuf (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e412e59 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:36 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) eam 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) p 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) 31648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x4003bdd82400 r (AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24) tatype 9 op 0 root 0 comm 0xaaab2a5a1f40 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24) jpbo-026-40:249633:249633 [0] NCCL INFO Broadcast: opCount 6e sendbuff 0x4003bdd82400 recvbuff (FSDPPolicyWorkerBase pid=364945) f 0x400eb9e88c00 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0 (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 128 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCo (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401325c00000 recvbuff 0x401760000000 count 12582912 data (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401328c00000 recvbuff 0x401760000000 count 12582912 datat (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401666000000 recvbuff 0x401666000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount af sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [ (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) ount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:4423 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) unt fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:44231 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount be sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x401660000000 recvbuff 0x401660000000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x40188e200000 recvbuff 0x401913c00000 count 1048576 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [n (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 1c8 sendbuff 0x400edfa88e00 r (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL IN (FSDPPolicyWorkerBase pid=364947) type 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c (FSDPPolicyWorkerBase pid=364944) ype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020 (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x4016d9800000 recvbuff 0x4016d9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCo (AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24) 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab2a5a1f40 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24) jpbo-026-40:249633:249633 [0] NCCL INFO Broadcast: opCount 8a s (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) FO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401336400000 recvbuff 0x4017aa400000 count 4194304 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=4] s (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opC (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount da sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount db sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) e5 sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x401660000000 recvbuff 0x401660000000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 128 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 128 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x401666000000 recvbuff 0x401666000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:44 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) tream 0x400e093f6c80 (FSDPPolicyWorkerBase pid=364947) jpbo-026 (FSDPPolicyWorkerBase pid=364944) ount 0 sendbuff 0x40133e400000 recvbuff 0x401766000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020 (FSDPPolicyWorkerBase pid=364944) jpbo-026- (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) ] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401339c00000 recvbuff 0x401760000000 count 1258 (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) unt d6 sendbuff 0x4003ddd82400 recvbuff 0x4003ddd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab1bf56550 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) jpbo-029-33:349999:349999 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab1bf56550 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161) jpbo-029-33:349999:349999 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab1bf56550 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161) endbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab1d4cfe60 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161) jpbo-029-33:350003:350003 [0] NCCL I (FSDPPolicyWorkerBase pid=364945) INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nrank (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 208 sendbuff 0x400edfa88e00 recvbuff 0x400e (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [nr (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 110 sendbuff 0x401666000000 recvb (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) 2912 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: op (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) uff 0x401666000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllGat (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) Count 18f sendbuff 0x400eb9e89000 recvbuff 0x400eb9e89000 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream ( (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce (AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165) jpbo-029-37:304985:304985 [0] NCCL INFO Broadcast: opCount 12b sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 c (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) NFO Broadcast: opCount a5 sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab063e0f80 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) jpbo-029 (AsyncVLLMInferenceEngine pid=478791, ip=10.128.25.176) jpbo-029 (AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24) jpbo-026 (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026 (AsyncVLLMInferenceEngine pid=379729, ip=10.128.25.25) jpbo-026 (AsyncVLLMInferenceEngine pid=379730, ip=10.128.25.25) jpbo-026 (AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164) jpbo-029 (AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164) jpbo-029 (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) jpbo-029 (AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174) jpbo-029 (AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163) jpbo-029 (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) jpbo-029 (AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161) jpbo-029 (AsyncVLLMInferenceEngine pid=349500, ip=10.128.25.161) jpbo-029 (AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23) jpbo-026 (AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23) jpbo-026 (AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162) jpbo-029 (AsyncVLLMInferenceEngine pid=300697, ip=10.128.25.162) jpbo-029 (AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28) jpbo-026 (AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28) jpbo-026 (AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165) jpbo-029 (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029 (AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27) jpbo-026 (AsyncVLLMInferenceEngine pid=326287, ip=10.128.25.27) jpbo-026 (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) Count fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953: (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:4 (FSDPPolicyWorkerBase pid=364946) her: opCount 0 sendbuff 0x400edfa7b200 recvbuff 0x400edfa8b000 count 1024 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400edfa7d200 recvbuff 0x400edfa88e00 count 1024 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] st (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) nil) (FSDPPolicyWorkerBase pid=364945) : opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (n (FSDPPolicyWorkerBase pid=364946) ream 0x400e067c71c0 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 248 sendbuff 0x400edfa88e00 recvbuff 0x400edfa88e00 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:44260 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401391400000 recvbuff 0x401760000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401374400000 recvbuff 0x401760000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks= (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllGath (AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176) omm 0xaaab25495c20 [nranks=49] stream (nil) (AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176) jpbo-029-48:479197:479197 [0] NCCL INFO Broadcast: opCount 156 sendbuff 0x4016d (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) jpbo-029-48:479189:479189 [0] NCCL INFO Broadcast: opCount da sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab063e0f80 [nran (FSDPPolicyWorkerBase pid=364945) il) (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e53e00 recvbuff 0x400eb9e8b000 count 1024 dataty (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) t 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo (FSDPPolicyWorkerBase pid=364947) j (FSDPPolicyWorkerBase pid=364944) er: opCount 0 sendbuff 0x400e99e53200 recvbuff 0x400e99e88e00 count 32 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020 (FSDPPolicyWorkerBase pid=364945) pe 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 265 sendbuff 0x400e9fa88c00 recv (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) buff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NC (FSDPPolicyWorkerBase pid=364946) 0 sendbuff 0x401900000000 recvbuff 0x4016a0000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0 (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401379800000 recvbuff 0x4017aa400000 count 1048576 datatype 9 op (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:443494 [0] NCCL INFO cudaDriverVersion 13020 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:443494 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.  [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.174<0> [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO Initialized NET plugin IB [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Assigned NET plugin IB to comm [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Using network IB [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO DMA-BUF is available on GPU device 0 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO ncclCommInitRankConfig comm 0x400ef44ecc80 rank 0 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init START [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> [repeated 47x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO comm 0x400ef44ecc80 rank 0 nRanks 49 nNodes 13 localRanks 1 localRank 0 MNNVL 0 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Trees [0] 26/-1/-1->0->-1 [1] 27/-1/-1->0->-1 [2] 28/-1/-1->0->-1 [3] 37/-1/-1->0->-1 [4] 41/-1/-1->0->29 [5] 42/-1/-1->0->30 [6] 43/-1/-1->0->31 [7] 44/-1/-1->0->32 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO P2P Chunksize set to 131072 [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) jpbo-029-46:443494:444068 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.  [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442612 [0] NCCL INFO [Proxy Service] Device 0 CPU core 51 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442613 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 52 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442614 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 54 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 07/0 : 0[0] -> 44[3] [send] via NET/IB/0 [repeated 282x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Channel 00/0 : 26[0] -> 0[0] [receive] via NET/IB/0 [repeated 279x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:250043 [0] NCCL INFO Channel 06/0 : 22[1] -> 21[0] via P2P/IPC [repeated 654x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:250071 [0] NCCL INFO Allocated shareable buffer 0x40177e600000 size 2097152 ipcDesc 0x40094433fc90 [repeated 1312x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) jpbo-026-40:249637:250044 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x4016dd800000 [repeated 1331x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27) jpbo-026-43:326764:327196 [0] NCCL INFO Allocated shareable buffer 0x40177e000000 size 2097152 (FSDPPolicyWorkerBase pid=364944) jp [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25) jpbo- (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Connected all trees [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO threadThresholds 8/8/64 | 392/8/64 | 512 | 512 [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 1 p2p channels per peer [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) jpbo-026-40:249637:250044 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO ncclCommInitRankConfig comm 0x400ef44ecc80 rank 0 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init COMPLETE [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442611 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 49 total 3.47 (kernels 0.00, alloc 0.00, bootstrap 2.79, allgathers 0.04, topo 0.01, graphs 0.00, connections 0.63, rest 0.00) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27) ipcDesc 0x40094433fc30 (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) CL INFO AllGather: opCount 0 sendbuff 0x4018ee800000 recvbuff 0x401660000000 count 12582912 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCo (FSDPPolicyWorkerBase pid=364946) 67c71c0 (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 288 sendbuff 0x400edfa88e00 recvbuff 0x400edfa88e00 count 1 datatyp (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520 (AsyncVLLMInferenceEngine pid=300569, ip=10.128.25.162) (EngineCore_DP0 pid=301120) INFO 06-12 01:05:34 [block_pool.py:452] Successfully reset prefix cache (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount ff sendbuff 0x400edfa88c00 recvbuff 0x400edfa88c00 count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x40138a400000 recvbuff 0x401760000000 count 155582464 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520 [repeated 3044x across cluster] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil) [repeated 1532x across cluster] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff  [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatyp (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opC [repeated 3x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:05:44] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:05:44] OK: RSS 0.89 GiB | node mem 381.3/858.0 GiB used (44.4%), avail 476.6 GiB (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) INFO Broadcast: opCount fa sendbuff 0x400e99e2b600 recvbuff 0x400e99e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x40114a400000 count 155582464 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream (nil) [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400e9fa64200 recvbuff 0x400e9fa88e00 count 1024 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] strea (AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28) tream (nil) [repeated 25x across cluster] (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: op [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) 9800000 recvbuff 0x4016d9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab0d165c60 [nranks=49] stream (nil) [repeated 31x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount a sendbuff 0x40079203c400 recvbuff 0x40079203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount b sendbuff 0x40079203c400 recvbuff 0x40079203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount c sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount f sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil) [repeated 98x across cluster] (FSDPPolicyWorkerBase pid=364944) 4700 [nranks=8] stream (nil) [repeated 88x across cluster] (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) m 0x400dfa7c8310 (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root (FSDPPolicyWorkerBase pid=364947) Count 0 sendbuff 0x40133b400000 recvbuff 0x401766000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160 [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount 2a sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nran [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164) jpbo-029-36:305976:305976 [0] NCCL INFO Broadcast: opCount 1c sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab0c5666e0 [nranks=49] s [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174) 0 comm 0xaaab132435a0 [nranks=49] stream (nil) [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x40138a000000 recvbuff 0x4017 [repeated 10x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4012e74000 (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root 0 comm 0x400e412e5990 [n (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) 66000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=4] stream 0x400e093f6c80 [repeated 10x across cluster] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCoun (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) jpbo-026-40:249637:249637 [0] NCCL INFO Broadcast: opCount 180 sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab0d165c60 [nranks=49] stream (nil) [repeated 14484x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil) [repeated 1323x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 1 [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=364945) 00 recvbuff 0x401766000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0 (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400e99e4e400 recvbuff 0x400e99e88c00 count 32 datatype 9 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount f6 sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root  [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 157 sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0 [repeated 25x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 22c sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x40 [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=364944) op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020 [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4018f0800000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfaa9fd0 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27) ecvbuff 0x40039dd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab0aad8260 [nranks=49] stream (nil) [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount d7 sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 12 [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164) jpbo-029-36:305976:305976 [0] NCCL INFO Broadcast: opCount 52 sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 da [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL  [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 o (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) 4] stream 0x400e093f6c80 [repeated 7x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=364946) INFO AllGather: opCount 0 sendbuff 0x4018b1000000 recvbuff 0x401389180000 count 1048576 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0 (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e412e [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) p 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) (FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172) 8 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil) [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27) jpbo-026-43:326770:326770 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x40039dd82400 r [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) tatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount 6e sendbuff 0x4003ddd82400 recvbuff  [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 25x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e50a00 recvbuff 0x400eb9e88e00 count 32 datat (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount af sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [ [repeated 23x across cluster] (FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172) ount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil) (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount be sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=364946) jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4018c2000000 recvbuff 0x401932a00000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06a9e080 [n (FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172) ype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520 (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27) jpbo-026-43:326770:326770 [0] NCCL INFO Broadcast: opCo [repeated 22x across cluster] (FSDPPolicyWorkerBase pid=364946) count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil) [repeated 26x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount 8a s [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount da sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil) [repeated 73x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount db sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (FSDPPolicyWorkerBase pid=364945) 8f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil) (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 98x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 48x across cluster] (AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24) jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil) [repeated 98x across cluster] (FSDPPolicyWorkerBase pid=364945) jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401342a00000 recvbuff 0x40171a400000 count (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) unt d6 sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4016d9800000 recvbuff 0x4016d9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x401739200000 recvbuff 0x401739200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil) [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) endbuff 0x4003ddd82400 recvbuff 0x4003ddd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab3afe1e00 [nranks=49] stream (nil) [repeated 24x across cluster] (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) jpbo-029-35:301662:301662 [0] NCCL I [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=364945) 1048576 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0 (FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172) jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: op (FSDPPolicyWorkerBase pid=364947) jpbo-026-33:364947:365414 [0] NCCL INFO AllGat (AsyncVLLMInferenceEngine pid=316639, ip=10.128.25.28) jpbo-026-44:317124:317124 [0] NCCL INFO Broadcast: opCount 12b sendbuff 0x401739200000 recvbuff 0x401739200000 count 4194304 datatype 9 op 0 root 0 c [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) NFO Broadcast: opCount a5 sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil) [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=364947) her: opCount 0 sendbuff 0x400e99e53200 recvbuff 0x400e99e88e00 count 32 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160 (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) omm 0xaaab5583a610 [nranks=49] stream (nil) [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165) jpbo-029-37:304986:304986 [0] NCCL INFO Broadcast: opCount 156 sendbuff 0x4016f [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165) jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount da sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nran [repeated 24x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype (FSDPPolicyWorkerBase pid=364945) jpbo (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) jpbo- [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) (EngineCore_DP0 pid=316475) INFO 06-12 01:05:34 [block_pool.py:452] Successfully reset prefix cache [repeated 47x across cluster] (FSDPPolicyWorkerBase pid=364944) jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4013a0000000 recvbuff 0x401760000000 count 155582464 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020 [repeated 42x across cluster] (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:05:51] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:05:51] OK: RSS 0.89 GiB | node mem 368.5/858.0 GiB used (42.9%), avail 489.5 GiB (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:05:58] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:05:58] OK: RSS 0.89 GiB | node mem 379.8/858.0 GiB used (44.3%), avail 478.2 GiB (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:06:07] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) [repeated 2x across cluster] (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:06:07] OK: RSS 0.89 GiB | node mem 389.9/858.0 GiB used (45.4%), avail 468.0 GiB [repeated 2x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:07:44] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:07:44] OK: RSS 0.89 GiB | node mem 381.5/858.0 GiB used (44.5%), avail 476.5 GiB (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:07:51] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:07:51] OK: RSS 0.89 GiB | node mem 368.5/858.0 GiB used (43.0%), avail 489.4 GiB (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:07:58] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:07:58] OK: RSS 0.89 GiB | node mem 379.9/858.0 GiB used (44.3%), avail 478.0 GiB (skyrl_entrypoint pid=364638) [fd-monitor] [01:08:01] OK: 104 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=364638) [fd-monitor] [01:08:01] OK: RSS 1.74 GiB | node mem 245.2/858.0 GiB used (28.6%), avail 612.8 GiB (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:08:07] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:08:07] OK: RSS 0.89 GiB | node mem 390.0/858.0 GiB used (45.5%), avail 468.0 GiB (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) Provider List: https://docs.litellm.ai/docs/providers [repeated 3x across cluster] (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) Provider List: https://docs.litellm.ai/docs/providers [repeated 714x across cluster] (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers [repeated 797x across cluster] (AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27) INFO 06-12 01:09:13 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:18 [loggers.py:259] Engine 000: Avg prompt throughput: 2.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164) INFO 06-12 01:09:17 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 9x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=350242, ip=10.128.25.161) (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0% [repeated 14x across cluster] (AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163) INFO 06-12 01:09:23 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 21x across cluster] (AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24) INFO 06-12 01:09:25 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% [repeated 29x across cluster] (AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162) INFO 06-12 01:09:31 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:38 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0% [repeated 30x across cluster] (AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164) INFO 06-12 01:09:34 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:43 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0% [repeated 28x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:09:44] OK: 263 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=350242, ip=10.128.25.161) [fd-monitor] [01:09:44] OK: RSS 1.13 GiB | node mem 382.1/858.0 GiB used (44.5%), avail 475.9 GiB (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0% [repeated 31x across cluster] (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:09:51] OK: 263 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=479438, ip=10.128.25.176) [fd-monitor] [01:09:51] OK: RSS 1.13 GiB | node mem 369.2/858.0 GiB used (43.0%), avail 488.8 GiB (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% [repeated 31x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:09:58 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% [repeated 28x across cluster] (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:09:58] OK: 215 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=301361, ip=10.128.25.162) [fd-monitor] [01:09:58] OK: RSS 1.07 GiB | node mem 380.3/858.0 GiB used (44.3%), avail 477.7 GiB (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:10:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0% [repeated 29x across cluster] (skyrl_entrypoint pid=364638) [fd-monitor] [01:10:01] OK: 177 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072) (skyrl_entrypoint pid=364638) [fd-monitor] [01:10:01] OK: RSS 1.76 GiB | node mem 245.3/858.0 GiB used (28.6%), avail 612.7 GiB (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:10:07] OK: 215 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072) (RolloutCoordinator pid=380389, ip=10.128.25.25) [fd-monitor] [01:10:07] OK: RSS 1.07 GiB | node mem 390.6/858.0 GiB used (45.5%), avail 467.4 GiB (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:10:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0% [repeated 28x across cluster] (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:10:13 [loggers.py:259] Engine 000: Avg prompt throughput: 190.6 tokens/s, Avg generation throughput: 26.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 64.5% [repeated 27x across cluster] (AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163) INFO 06-12 01:10:16 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) (RolloutCoordinator pid=479438, ip=10.128.25.176) Provider List: https://docs.litellm.ai/docs/providers (RolloutCoordinator pid=479438, ip=10.128.25.176) (AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23) INFO 06-12 01:10:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 64.5% [repeated 30x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (RolloutCoordinator pid=350242, ip=10.128.25.161) (AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25) INFO 06-12 01:10:18 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 4x across cluster] (RolloutCoordinator pid=350242, ip=10.128.25.161) Provider List: https://docs.litellm.ai/docs/providers [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:23 [loggers.py:259] Engine 000: Avg prompt throughput: 57.1 tokens/s, Avg generation throughput: 83.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 80.3% [repeated 27x across cluster] (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (RolloutCoordinator pid=380389, ip=10.128.25.25) (AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174) INFO 06-12 01:10:27 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) (RolloutCoordinator pid=301361, ip=10.128.25.162) Provider List: https://docs.litellm.ai/docs/providers [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 56.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 80.3% [repeated 26x across cluster] (AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28) INFO 06-12 01:10:31 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 80.3% [repeated 35x across cluster] (AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176) INFO 06-12 01:10:34 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:38 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 80.3% [repeated 41x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:43 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 80.3% [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 80.3% [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24) INFO 06-12 01:10:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 80.3% [repeated 38x across cluster] (AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174) INFO 06-12 01:10:56 [loggers.py:259] Engine 000: Avg prompt throughput: 64.7 tokens/s, Avg generation throughput: 77.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 68.0% [repeated 21x across cluster] Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global', 'trainer.algorithm.use_tis=true', 'trainer.algorithm.tis_imp_ratio_cap=2.0', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/explore-tis-untrunc', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=explore-tis-untrunc', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.collect_rollout_details=true', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true', 'generator.sampling_params.temperature=1.0', 'generator.sampling_params.top_p=1.0', 'generator.sampling_params.top_k=-1'] Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in main() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() ^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( ^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run _ = ret.return_value ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) ^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main ray.get(skyrl_entrypoint.remote(cfg)) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get values, debugger_breakpoint = worker.get_objects( ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1017, in get_objects raise value ray.exceptions.NodeDiedError: Task failed because the node it was running on is dead or unavailable. Node IP: 10.128.25.17, node ID: 96706d0fc7d05b397467667c5e336ac21712cf7e0c8bf1b2f4037765. This can happen if the node was preempted, had a hardware failure, or its raylet crashed unexpectedly. To see node death information, use `ray list nodes --filter node_id=96706d0fc7d05b397467667c5e336ac21712cf7e0c8bf1b2f4037765`, check the Ray dashboard cluster page, search the node ID in the GCS logs, or use `ray logs raylet.out -ip 10.128.25.17`. Stopping Ray cluster... Ray cluster stopped [RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...