7560 lines
835 KiB
Plaintext
7560 lines
835 KiB
Plaintext
|
||
The following have been reloaded with a version change:
|
||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||
|
||
|
||
Lmod is automatically replacing "GCC/14.3.0" with
|
||
"nvidia-compilers/25.9-CUDA-13".
|
||
|
||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
[ray] RAY_TMPDIR=/tmp/ray/ray_736687
|
||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_736687
|
||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_736687
|
||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||
[proxy] Tunnel port: 7003
|
||
[proxy] Node IP: 10.128.25.17 (workers will connect here)
|
||
[proxy] ✓ SSH tunnel started successfully
|
||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_736687.conf
|
||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||
[proxy] ✓ Daytona timeout settings configured
|
||
[proxy] Testing proxy connectivity...
|
||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_736687.conf
|
||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||
[proxy] ✓ Tunnel accessible at 10.128.25.17:7003 (workers can connect)
|
||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||
=== Universal RL Training Runner ===
|
||
Config: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/configs/explore-tis-untrunc_rl_config.json
|
||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Python version: Python 3.12.12
|
||
UV_USE_IO_URING: 0
|
||
Proxy: DISABLED (direct internet or not configured)
|
||
========================================
|
||
=== RLJobRunner: explore-tis-untrunc ===
|
||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/wandb
|
||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/wandb
|
||
HF_TOKEN=****pDbg
|
||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||
Environment configured:
|
||
TENSOR_PARALLEL_SIZE=1
|
||
NUM_INFERENCE_ENGINES=56
|
||
POLICY_NUM_NODES=14
|
||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/wandb
|
||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||
Cleaning up existing Ray instances...
|
||
=== Starting Ray Cluster ===
|
||
Nodes: 14
|
||
GPUs per node: 4
|
||
CPUs per node: 288
|
||
Head node: jpbo-026-33 (10.128.25.17)
|
||
Ray port: 6379
|
||
============================
|
||
Starting Ray head on jpbo-026-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-026-33.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.17 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.25.17 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray head on jpbo-026-33
|
||
Starting Ray worker on jpbo-026-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-39.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.23 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.23 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 1 on jpbo-026-39
|
||
Starting Ray worker on jpbo-026-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-40.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.24 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.24 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 2 on jpbo-026-40
|
||
Starting Ray worker on jpbo-026-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-41.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.25 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.25 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 3 on jpbo-026-41
|
||
Starting Ray worker on jpbo-026-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-43.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.27 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.27 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 4 on jpbo-026-43
|
||
Starting Ray worker on jpbo-026-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-026-44.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.28 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-026-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.28 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 5 on jpbo-026-44
|
||
Starting Ray worker on jpbo-029-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-33.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.161 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.161 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 6 on jpbo-029-33
|
||
Starting Ray worker on jpbo-029-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-34.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.162 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.162 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 7 on jpbo-029-34
|
||
Starting Ray worker on jpbo-029-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-35.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.163 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.163 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 8 on jpbo-029-35
|
||
Starting Ray worker on jpbo-029-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-36.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.164 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.164 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 9 on jpbo-029-36
|
||
Starting Ray worker on jpbo-029-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-37.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.165 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.165 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 10 on jpbo-029-37
|
||
Starting Ray worker on jpbo-029-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-44.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.172 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.172 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 11 on jpbo-029-44
|
||
Starting Ray worker on jpbo-029-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-46.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.174 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.174 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 12 on jpbo-029-46
|
||
Starting Ray worker on jpbo-029-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-029-48.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.25.176 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-029-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.25.17:6379 --node-ip-address=10.128.25.176 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 13 on jpbo-029-48
|
||
Waiting for cluster (56 GPUs, 14 nodes)...
|
||
Connecting to Ray at 10.128.25.17:6379 (expecting 14 nodes, 56.0 GPUs)
|
||
Ray connection established, polling for resources...
|
||
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'CPU': 4032.0, 'object_store_memory': 601295421440.0, 'memory': 10860264161280.0, 'GPU': 56.0, 'accelerator_type:GH200': 14.0, 'node:10.128.25.161': 1.0, 'node:10.128.25.176': 1.0, 'node:10.128.25.162': 1.0, 'node:10.128.25.25': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.25.17': 1.0, 'node:10.128.25.23': 1.0, 'node:10.128.25.164': 1.0, 'node:10.128.25.165': 1.0, 'node:10.128.25.24': 1.0, 'node:10.128.25.172': 1.0, 'node:10.128.25.174': 1.0, 'node:10.128.25.28': 1.0, 'node:10.128.25.163': 1.0, 'node:10.128.25.27': 1.0}
|
||
✓ Ray cluster ready
|
||
=== Ray Cluster Ready ===
|
||
Address: 10.128.25.17:6379
|
||
Total GPUs: 56
|
||
=========================
|
||
Ray cluster ready at 10.128.25.17:6379
|
||
Total GPUs available: 56
|
||
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
|
||
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
|
||
|
||
Running SkyRL:
|
||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
|
||
Args: 126 Hydra arguments
|
||
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
|
||
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
|
||
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global trainer.algorithm.use_tis=true trainer.algorithm.tis_imp_ratio_cap=2.0 trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/explore-tis-untrunc ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=explore-tis-untrunc trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.collect_rollout_details=true +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true generator.sampling_params.temperature=1.0 generator.sampling_params.top_p=1.0 generator.sampling_params.top_k=-1
|
||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_736687.conf
|
||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
2026-06-12 00:53:52.012 | WARNING | skyrl_train.utils.utils:validate_cfg:479 - `generator.sampling_params.logprobs` is `None` but `trainer.algorithm.use_tis` is `True`. Setting `logprobs` to `True`.
|
||
2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:833 - Exporting wandb api key to ray runtime env
|
||
2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:852 - Exporting RAY_ADDRESS to ray runtime env
|
||
2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:877 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
|
||
2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:877 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
|
||
2026-06-12 00:53:52.180 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:877 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
|
||
2026-06-12 00:53:52,180 INFO worker.py:1680 -- Using address 10.128.25.17:6379 set in the environment variable RAY_ADDRESS
|
||
2026-06-12 00:53:52,214 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.25.17:6379...
|
||
2026-06-12 00:53:52,222 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
2026-06-12 00:53:52,225 INFO worker.py:2007 -- Connected to Ray cluster.
|
||
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||
warnings.warn(
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(pid=348050)[0m [2026-06-12 00:53:52,634 E 348050 349258] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[33m(raylet)[0m 2026-06-12 00:53:53,525 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
2026-06-12 00:53:54.651 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||
[36m(pid=348068)[0m [2026-06-12 00:53:58,380 E 348068 349758] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 8x across cluster][0m
|
||
[33m(raylet)[0m 2026-06-12 00:53:55,365 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 2x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:01.116[0m | [1mINFO [0m | [36mskyrl_train.entrypoints.main_base[0m:[36m_configure_log_level[0m:[36m214[0m - [1mSkyRL log level set to: INFO[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:01.404[0m | [1mINFO [0m | [36mexamples.terminal_bench.dataset[0m:[36m_load_data_files[0m:[36m40[0m - [1mLoading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:04.195[0m | [1mINFO [0m | [36mexamples.terminal_bench.dataset[0m:[36m_load_data_files[0m:[36m50[0m - [1mFound 5000 valid task directories out of 5000 total directories[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:04.195[0m | [1mINFO [0m | [36mexamples.terminal_bench.dataset[0m:[36m__init__[0m:[36m27[0m - [1mTerminalBenchTaskDataset initialized with 5000 task paths[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:04.206[0m | [1mINFO [0m | [36mskyrl_train.entrypoints.main_base[0m:[36m_setup_trainer[0m:[36m407[0m - [1mdata:
|
||
[36m(skyrl_entrypoint pid=364638)[0m train_data:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
|
||
[36m(skyrl_entrypoint pid=364638)[0m val_data: []
|
||
[36m(skyrl_entrypoint pid=364638)[0m trainer:
|
||
[36m(skyrl_entrypoint pid=364638)[0m placement:
|
||
[36m(skyrl_entrypoint pid=364638)[0m colocate_all: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m colocate_policy_ref: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_num_nodes: 2
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_num_gpus_per_node: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m critic_num_nodes: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m critic_num_gpus_per_node: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m ref_num_nodes: 2
|
||
[36m(skyrl_entrypoint pid=364638)[0m ref_num_gpus_per_node: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_strict_spread_pg: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_per_gpu_bundles: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_force_cvd_mask: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m sequence_parallel_backend: ulysses
|
||
[36m(skyrl_entrypoint pid=364638)[0m strategy: fsdp2
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy:
|
||
[36m(skyrl_entrypoint pid=364638)[0m model:
|
||
[36m(skyrl_entrypoint pid=364638)[0m path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
|
||
[36m(skyrl_entrypoint pid=364638)[0m lora:
|
||
[36m(skyrl_entrypoint pid=364638)[0m rank: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m alpha: 16
|
||
[36m(skyrl_entrypoint pid=364638)[0m dropout: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m lora_sync_path: /tmp/skyrl_lora_sync
|
||
[36m(skyrl_entrypoint pid=364638)[0m target_modules: all-linear
|
||
[36m(skyrl_entrypoint pid=364638)[0m exclude_modules: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepspeed_config: ${deepspeed_config.train}
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer: AdamW
|
||
[36m(skyrl_entrypoint pid=364638)[0m lr: 8.0e-06
|
||
[36m(skyrl_entrypoint pid=364638)[0m adam_betas:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - 0.9
|
||
[36m(skyrl_entrypoint pid=364638)[0m - 0.999
|
||
[36m(skyrl_entrypoint pid=364638)[0m weight_decay: 0.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_grad_norm: 0.9
|
||
[36m(skyrl_entrypoint pid=364638)[0m offload_after_step: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m num_warmup_steps: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m scheduler: constant_with_warmup
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m fsdp_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m reshard_after_forward: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m fsdp_size: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_token_dispatcher_type: alltoall
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_router_replay: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_grouped_gemm: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m ep_comm_backend: torch
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepep_num_sms: 20
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepep_token_chunk_size: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m sequence_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_torch_compile: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m record_memory: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m megatron_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m tensor_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m pipeline_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m context_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_tensor_parallel_size: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m ddp_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m grad_reduce_in_fp32: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m overlap_grad_reduce: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m overlap_param_gather: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m average_in_collective: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m model_config_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m torch_profiler_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m ranks: []
|
||
[36m(skyrl_entrypoint pid=364638)[0m save_path: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_config_kwargs:
|
||
[36m(skyrl_entrypoint pid=364638)[0m overlap_cpu_optimizer_d2h_h2d: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_precision_aware_optimizer: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_offload_fraction: 0.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m transformer_config_kwargs:
|
||
[36m(skyrl_entrypoint pid=364638)[0m recompute_granularity: full
|
||
[36m(skyrl_entrypoint pid=364638)[0m recompute_modules:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - core_attn
|
||
[36m(skyrl_entrypoint pid=364638)[0m recompute_method: uniform
|
||
[36m(skyrl_entrypoint pid=364638)[0m recompute_num_layers: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m empty_cuda_cache: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m ref:
|
||
[36m(skyrl_entrypoint pid=364638)[0m model:
|
||
[36m(skyrl_entrypoint pid=364638)[0m path: ${trainer.policy.model.path}
|
||
[36m(skyrl_entrypoint pid=364638)[0m sequence_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepspeed_config: ${deepspeed_config.eval}
|
||
[36m(skyrl_entrypoint pid=364638)[0m fsdp_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m reshard_after_forward: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m fsdp_size: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_token_dispatcher_type: alltoall
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_router_replay: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_grouped_gemm: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m ep_comm_backend: torch
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepep_num_sms: 20
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepep_token_chunk_size: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m megatron_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m tensor_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m pipeline_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m context_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m model_config_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m transformer_config_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m critic:
|
||
[36m(skyrl_entrypoint pid=364638)[0m model:
|
||
[36m(skyrl_entrypoint pid=364638)[0m path: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m lora:
|
||
[36m(skyrl_entrypoint pid=364638)[0m rank: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m alpha: 16
|
||
[36m(skyrl_entrypoint pid=364638)[0m dropout: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m target_modules: all-linear
|
||
[36m(skyrl_entrypoint pid=364638)[0m exclude_modules: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepspeed_config: ${deepspeed_config.train}
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer: AdamW
|
||
[36m(skyrl_entrypoint pid=364638)[0m lr: 5.0e-06
|
||
[36m(skyrl_entrypoint pid=364638)[0m adam_betas:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - 0.9
|
||
[36m(skyrl_entrypoint pid=364638)[0m - 0.999
|
||
[36m(skyrl_entrypoint pid=364638)[0m weight_decay: 0.01
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_grad_norm: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m offload_after_step: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m num_warmup_steps: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m scheduler: constant_with_warmup
|
||
[36m(skyrl_entrypoint pid=364638)[0m optimizer_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m fsdp_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m cpu_offload: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m reshard_after_forward: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m fsdp_size: -1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_model_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m expert_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_token_dispatcher_type: alltoall
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_router_replay: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m moe_grouped_gemm: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m ep_comm_backend: torch
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepep_num_sms: 20
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepep_token_chunk_size: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m sequence_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m algorithm:
|
||
[36m(skyrl_entrypoint pid=364638)[0m advantage_estimator: rloo_n
|
||
[36m(skyrl_entrypoint pid=364638)[0m kl_ctrl:
|
||
[36m(skyrl_entrypoint pid=364638)[0m type: fixed
|
||
[36m(skyrl_entrypoint pid=364638)[0m kl_target: 0.1
|
||
[36m(skyrl_entrypoint pid=364638)[0m horizon: 10000
|
||
[36m(skyrl_entrypoint pid=364638)[0m kl_estimator_type: k3
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_kl_estimator_k3: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_abs_kl: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_kl_in_reward: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_kl_loss: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m kl_loss_coef: 0.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_entropy_loss: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m entropy_loss_coef: 0.01
|
||
[36m(skyrl_entrypoint pid=364638)[0m advantage_batch_normalize: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m value_head_prefix: value_head
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_loss_type: regular
|
||
[36m(skyrl_entrypoint pid=364638)[0m loss_reduction: seq_mean_token_sum_norm_global
|
||
[36m(skyrl_entrypoint pid=364638)[0m global_loss_denom: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m grpo_norm_by_std: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m rloo_n_min_group_size: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m rloo_n_filter_zero_reward_groups: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m lambd: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m gamma: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m eps_clip_low: 0.2
|
||
[36m(skyrl_entrypoint pid=364638)[0m eps_clip_high: 0.05
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_ratio_c: 3.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m tis_imp_ratio_cap: 2.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_tis: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m sapo:
|
||
[36m(skyrl_entrypoint pid=364638)[0m tau_pos: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m tau_neg: 1.05
|
||
[36m(skyrl_entrypoint pid=364638)[0m value_clip: 0.2
|
||
[36m(skyrl_entrypoint pid=364638)[0m dynamic_sampling:
|
||
[36m(skyrl_entrypoint pid=364638)[0m type: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_sample_batches: 30
|
||
[36m(skyrl_entrypoint pid=364638)[0m min_replace_ratio: 0.3
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_cov:
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_ratio: 0.0002
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_cov_lb: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_cov_ub: 5.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m kl_cov:
|
||
[36m(skyrl_entrypoint pid=364638)[0m kl_cov_frac: 0.2
|
||
[36m(skyrl_entrypoint pid=364638)[0m ppo_kl_coef: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m cispo:
|
||
[36m(skyrl_entrypoint pid=364638)[0m cispo_eps_clip_low: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m cispo_eps_clip_high: 5
|
||
[36m(skyrl_entrypoint pid=364638)[0m z_clip:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m alpha: 0.97
|
||
[36m(skyrl_entrypoint pid=364638)[0m z_thresh: 2.5
|
||
[36m(skyrl_entrypoint pid=364638)[0m warmup_steps: 25
|
||
[36m(skyrl_entrypoint pid=364638)[0m mode: zscore
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_option: adaptive_scaling
|
||
[36m(skyrl_entrypoint pid=364638)[0m clip_factor: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m skip_update_on_spike: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m stale_clip:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m alpha: 0.3
|
||
[36m(skyrl_entrypoint pid=364638)[0m entropy_threshold: 0.15
|
||
[36m(skyrl_entrypoint pid=364638)[0m entropy_window: 10
|
||
[36m(skyrl_entrypoint pid=364638)[0m min_lr_scale: 0.1
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_seq_len: 1004095
|
||
[36m(skyrl_entrypoint pid=364638)[0m fully_async:
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_staleness_steps: 16
|
||
[36m(skyrl_entrypoint pid=364638)[0m num_parallel_generation_workers: 338
|
||
[36m(skyrl_entrypoint pid=364638)[0m gradient_checkpointing: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m gradient_checkpointing_use_reentrant: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m seed: 42
|
||
[36m(skyrl_entrypoint pid=364638)[0m resume_mode: latest
|
||
[36m(skyrl_entrypoint pid=364638)[0m resume_path: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m ckpt_path: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_ckpts_to_keep: -1
|
||
[36m(skyrl_entrypoint pid=364638)[0m ckpt_interval: 2
|
||
[36m(skyrl_entrypoint pid=364638)[0m hf_save_interval: 5
|
||
[36m(skyrl_entrypoint pid=364638)[0m hf_upload_mode: latest
|
||
[36m(skyrl_entrypoint pid=364638)[0m export_path: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports
|
||
[36m(skyrl_entrypoint pid=364638)[0m bf16: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m epochs: 2
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_steps: 80
|
||
[36m(skyrl_entrypoint pid=364638)[0m update_epochs_per_batch: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m train_batch_size: 64
|
||
[36m(skyrl_entrypoint pid=364638)[0m policy_mini_batch_size: 64
|
||
[36m(skyrl_entrypoint pid=364638)[0m critic_mini_batch_size: 256
|
||
[36m(skyrl_entrypoint pid=364638)[0m micro_train_batch_size_per_gpu: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m micro_forward_batch_size_per_gpu: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m update_ref_every_epoch: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_sample_packing: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m eval_batch_size: 64
|
||
[36m(skyrl_entrypoint pid=364638)[0m eval_before_train: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m eval_interval: 999999
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_prompt_length: 999999
|
||
[36m(skyrl_entrypoint pid=364638)[0m flash_attn: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m disable_fast_tokenizer: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m target_modules: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m exclude_modules: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m project_name: OpenThoughts-Agent
|
||
[36m(skyrl_entrypoint pid=364638)[0m run_name: explore-tis-untrunc
|
||
[36m(skyrl_entrypoint pid=364638)[0m logger: console
|
||
[36m(skyrl_entrypoint pid=364638)[0m tracker_commit_each_step: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m dump_data_batch: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m dump_eval_results: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m log_level: INFO
|
||
[36m(skyrl_entrypoint pid=364638)[0m rope_scaling: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m rope_theta: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m step_wise_training: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m hf_hub_repo_id: laion/explore-tis-untrunc
|
||
[36m(skyrl_entrypoint pid=364638)[0m hf_hub_private: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m hf_hub_revision: main
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_db_registration: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m generator:
|
||
[36m(skyrl_entrypoint pid=364638)[0m model_name: ${trainer.policy.model.path}
|
||
[36m(skyrl_entrypoint pid=364638)[0m model_dtype: bfloat16
|
||
[36m(skyrl_entrypoint pid=364638)[0m timeout_multiplier: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m run_engines_locally: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m num_inference_engines: 48
|
||
[36m(skyrl_entrypoint pid=364638)[0m backend: vllm
|
||
[36m(skyrl_entrypoint pid=364638)[0m weight_sync_backend: nccl
|
||
[36m(skyrl_entrypoint pid=364638)[0m fuse_weights: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m weight_transfer_threshold_cuda_ipc_GB: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_pipeline_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_expert_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_data_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_mp_backend: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m n_samples_per_prompt: 8
|
||
[36m(skyrl_entrypoint pid=364638)[0m async_engine: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m batched: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_input_length: ${trainer.max_prompt_length}
|
||
[36m(skyrl_entrypoint pid=364638)[0m vllm_v1_disable_multiproc: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_prefix_caching: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_chunked_prefill: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_num_batched_tokens: 65536
|
||
[36m(skyrl_entrypoint pid=364638)[0m enforce_eager: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m fully_sharded_loras: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_ray_prometheus_stats: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m vllm_stats_interval: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m gpu_memory_utilization: 0.75
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_num_seqs: 24
|
||
[36m(skyrl_entrypoint pid=364638)[0m remote_inference_engine_urls:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - 127.0.0.1:8001
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_http_endpoint: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m http_endpoint_host: 127.0.0.1
|
||
[36m(skyrl_entrypoint pid=364638)[0m http_endpoint_port: 8000
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_turns: 999999
|
||
[36m(skyrl_entrypoint pid=364638)[0m chat_template:
|
||
[36m(skyrl_entrypoint pid=364638)[0m source: name
|
||
[36m(skyrl_entrypoint pid=364638)[0m name_or_path: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m chat_template_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m engine_init_kwargs:
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_model_len: 32768
|
||
[36m(skyrl_entrypoint pid=364638)[0m custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(skyrl_entrypoint pid=364638)[0m served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
|
||
[36m(skyrl_entrypoint pid=364638)[0m override_existing_update_group: disable
|
||
[36m(skyrl_entrypoint pid=364638)[0m sampling_params:
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_generate_length: 4096
|
||
[36m(skyrl_entrypoint pid=364638)[0m repetition_penalty: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m temperature: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m top_p: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m min_p: 0.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m top_k: -1
|
||
[36m(skyrl_entrypoint pid=364638)[0m logprobs: 0
|
||
[36m(skyrl_entrypoint pid=364638)[0m stop: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m use_conversation_multi_turn: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m append_eos_token_after_stop_str_in_multi_turn: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m eval_sampling_params:
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_generate_length: ${generator.sampling_params.max_generate_length}
|
||
[36m(skyrl_entrypoint pid=364638)[0m repetition_penalty: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m temperature: 0.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m top_p: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m min_p: 0.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m top_k: -1
|
||
[36m(skyrl_entrypoint pid=364638)[0m logprobs: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m stop: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m eval_n_samples_per_prompt: 8
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_reward_on_non_stop: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m apply_overlong_filtering: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m rope_scaling: ${trainer.rope_scaling}
|
||
[36m(skyrl_entrypoint pid=364638)[0m rope_theta: ${trainer.rope_theta}
|
||
[36m(skyrl_entrypoint pid=364638)[0m teacher:
|
||
[36m(skyrl_entrypoint pid=364638)[0m model_path: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m top_k_logprobs: 256
|
||
[36m(skyrl_entrypoint pid=364638)[0m num_inference_engines: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_tensor_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m inference_engine_pipeline_parallel_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m gpu_memory_utilization: 0.9
|
||
[36m(skyrl_entrypoint pid=364638)[0m enforce_eager: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m backend: vllm
|
||
[36m(skyrl_entrypoint pid=364638)[0m engine_init_kwargs: {}
|
||
[36m(skyrl_entrypoint pid=364638)[0m environment:
|
||
[36m(skyrl_entrypoint pid=364638)[0m env_class: gsm8k
|
||
[36m(skyrl_entrypoint pid=364638)[0m skyrl_gym:
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_env_workers: 32
|
||
[36m(skyrl_entrypoint pid=364638)[0m text2sql:
|
||
[36m(skyrl_entrypoint pid=364638)[0m db_path: /home/ray/default/sql_data
|
||
[36m(skyrl_entrypoint pid=364638)[0m llm_as_a_judge:
|
||
[36m(skyrl_entrypoint pid=364638)[0m model: gpt-4o-mini
|
||
[36m(skyrl_entrypoint pid=364638)[0m base_url: null
|
||
[36m(skyrl_entrypoint pid=364638)[0m search:
|
||
[36m(skyrl_entrypoint pid=364638)[0m log_requests: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m search_url: http://127.0.0.1:8000/retrieve
|
||
[36m(skyrl_entrypoint pid=364638)[0m topk: 3
|
||
[36m(skyrl_entrypoint pid=364638)[0m timeout: 30
|
||
[36m(skyrl_entrypoint pid=364638)[0m rollout:
|
||
[36m(skyrl_entrypoint pid=364638)[0m fanout:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m num_coordinators: 4
|
||
[36m(skyrl_entrypoint pid=364638)[0m cpus_per_coordinator: 8
|
||
[36m(skyrl_entrypoint pid=364638)[0m deepspeed_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m train:
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_optimization:
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage: 3
|
||
[36m(skyrl_entrypoint pid=364638)[0m offload_param:
|
||
[36m(skyrl_entrypoint pid=364638)[0m device: none
|
||
[36m(skyrl_entrypoint pid=364638)[0m offload_optimizer:
|
||
[36m(skyrl_entrypoint pid=364638)[0m device: none
|
||
[36m(skyrl_entrypoint pid=364638)[0m pin_memory: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m sub_group_size: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m reduce_bucket_size: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage3_param_persistence_threshold: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage3_prefetch_bucket_size: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage3_max_live_parameters: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage3_max_reuse_distance: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m round_robin_gradients: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_hpz_partition_size: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_quantized_weights: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_quantized_gradients: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m torch_autocast:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m dtype: bfloat16
|
||
[36m(skyrl_entrypoint pid=364638)[0m disable_trace_cache: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m data_types:
|
||
[36m(skyrl_entrypoint pid=364638)[0m grad_accum_dtype: fp32
|
||
[36m(skyrl_entrypoint pid=364638)[0m gradient_clipping: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m wall_clock_breakdown: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m prescale_gradient: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m eval:
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_optimization:
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage: 3
|
||
[36m(skyrl_entrypoint pid=364638)[0m stage3_param_persistence_threshold: auto
|
||
[36m(skyrl_entrypoint pid=364638)[0m offload_param:
|
||
[36m(skyrl_entrypoint pid=364638)[0m device: cpu
|
||
[36m(skyrl_entrypoint pid=364638)[0m pin_memory: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m torch_autocast:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m dtype: bfloat16
|
||
[36m(skyrl_entrypoint pid=364638)[0m gradient_clipping: 1.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m prescale_gradient: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m wall_clock_breakdown: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m terminal_bench_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m trials_dir: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/trace_jobs
|
||
[36m(skyrl_entrypoint pid=364638)[0m harbor:
|
||
[36m(skyrl_entrypoint pid=364638)[0m name: terminus-2
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_episodes: 999999
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_summarize: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m store_all_messages: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m trajectory_config:
|
||
[36m(skyrl_entrypoint pid=364638)[0m raw_content: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_episode_logging: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m record_terminal_session: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_pane_logging: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m strict_json_parser: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m interleaved_thinking: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m extra_body:
|
||
[36m(skyrl_entrypoint pid=364638)[0m chat_template_kwargs:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_thinking: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m override_timeout_sec: 900
|
||
[36m(skyrl_entrypoint pid=364638)[0m override_cpus: 1
|
||
[36m(skyrl_entrypoint pid=364638)[0m override_memory_mb: 2048
|
||
[36m(skyrl_entrypoint pid=364638)[0m override_storage_mb: 2048
|
||
[36m(skyrl_entrypoint pid=364638)[0m auto_snapshot: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m verifier_override_timeout_sec: 120
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_retries: 3
|
||
[36m(skyrl_entrypoint pid=364638)[0m min_wait_sec: 60.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_wait_sec: 600.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m wait_multiplier: 2.0
|
||
[36m(skyrl_entrypoint pid=364638)[0m exclude_exceptions:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - VerifierTimeoutError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - VerifierRuntimeError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - RewardFileNotFoundError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - RewardFileEmptyError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - VerifierOutputParseError
|
||
[36m(skyrl_entrypoint pid=364638)[0m n_concurrent_trials: 675
|
||
[36m(skyrl_entrypoint pid=364638)[0m log_level: INFO
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_reward_shaping: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m collect_rollout_details: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m enable_error_classification: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m mask_exceptions:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - DaytonaError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - EnvironmentStartTimeoutError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - NetworkError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - ConnectionError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - RewardFileNotFoundError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - RewardFileEmptyError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - AgentEnvironmentTimeoutError
|
||
[36m(skyrl_entrypoint pid=364638)[0m - ContextLengthExceededError
|
||
[36m(skyrl_entrypoint pid=364638)[0m default_error_treatment: zero
|
||
[36m(skyrl_entrypoint pid=364638)[0m passthrough_exceptions:
|
||
[36m(skyrl_entrypoint pid=364638)[0m - AgentTimeoutError
|
||
[36m(skyrl_entrypoint pid=364638)[0m zero_exceptions: []
|
||
[36m(skyrl_entrypoint pid=364638)[0m model_info:
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_input_tokens: 32000
|
||
[36m(skyrl_entrypoint pid=364638)[0m max_output_tokens: 4096
|
||
[36m(skyrl_entrypoint pid=364638)[0m archiving:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: false
|
||
[36m(skyrl_entrypoint pid=364638)[0m trace_upload:
|
||
[36m(skyrl_entrypoint pid=364638)[0m enabled: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m repo_org: DCAgent
|
||
[36m(skyrl_entrypoint pid=364638)[0m episodes: last
|
||
[36m(skyrl_entrypoint pid=364638)[0m dataset_type: SFT
|
||
[36m(skyrl_entrypoint pid=364638)[0m cleanup: true
|
||
[36m(skyrl_entrypoint pid=364638)[0m [0m
|
||
[36m(pid=348197)[0m [2026-06-12 00:54:01,724 E 348197 363053] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 268x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(skyrl_entrypoint pid=364638)[0m No module named 'vllm._version'
|
||
[36m(skyrl_entrypoint pid=364638)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[36m(skyrl_entrypoint pid=364638)[0m W0612 00:54:09.087000 364638 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[33m(raylet)[0m 2026-06-12 00:54:12,109 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
[33m(raylet, ip=10.128.25.23)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 41x across cluster][0m
|
||
[33m(raylet, ip=10.128.25.23)[0m 2026-06-12 00:54:16,191 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 6x across cluster][0m
|
||
[36m(pid=478659, ip=10.128.25.176)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(pid=478659, ip=10.128.25.176)[0m No module named 'vllm._version'
|
||
[36m(pid=478659, ip=10.128.25.176)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[33m(raylet, ip=10.128.25.161)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 78x across cluster][0m
|
||
[2026-06-12 00:54:22,401 E 364224 364547] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[33m(raylet, ip=10.128.25.162)[0m 2026-06-12 00:54:21,909 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 18x across cluster][0m
|
||
[36m(pid=316063, ip=10.128.25.23)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
|
||
[36m(pid=316063, ip=10.128.25.23)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
|
||
[36m(pid=316063, ip=10.128.25.23)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 3x across cluster][0m
|
||
[36m(RegistryActor pid=364573)[0m [2026-06-12 00:54:23,544 E 364573 364613] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[33m(raylet, ip=10.128.25.163)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 90x across cluster][0m
|
||
[33m(raylet, ip=10.128.25.174)[0m 2026-06-12 00:54:27,918 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 12x across cluster][0m
|
||
[36m(pid=249226, ip=10.128.25.24)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 20x across cluster][0m
|
||
[36m(pid=249226, ip=10.128.25.24)[0m No module named 'vllm._version'[32m [repeated 20x across cluster][0m
|
||
[36m(pid=249226, ip=10.128.25.24)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 20x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [2026-06-12 00:54:25,383 E 364638 364681] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 2x across cluster][0m
|
||
[33m(raylet, ip=10.128.25.165)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 56x across cluster][0m
|
||
[33m(raylet, ip=10.128.25.28)[0m 2026-06-12 00:54:32,714 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 9x across cluster][0m
|
||
[36m(pid=443231, ip=10.128.25.174)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 4x across cluster][0m
|
||
[36m(pid=443231, ip=10.128.25.174)[0m No module named 'vllm._version'[32m [repeated 4x across cluster][0m
|
||
[36m(pid=443231, ip=10.128.25.174)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 4x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [2026-06-12 00:54:36] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
|
||
[36m(skyrl_entrypoint pid=364638)[0m [2026-06-12 00:54:36] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
|
||
[36m(skyrl_entrypoint pid=364638)[0m [2026-06-12 00:54:37] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.317[0m | [1mINFO [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m_spin_up_http_endpoint[0m:[36m969[0m - [1mInferenceEngineClient HTTP endpoint started on 127.0.0.1:8000[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.317[0m | [1mINFO [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m__init__[0m:[36m61[0m - [1mInferenceEngineClient initialized with 48 engines.[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.319[0m | [1mINFO [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m_configure_harbor_logging[0m:[36m242[0m - [1mHarbor logging level set to INFO[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.320[0m | [1mINFO [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m__init__[0m:[36m142[0m - [1mTerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.321[0m | [1mINFO [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m__init__[0m:[36m158[0m - [1mTerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.322[0m | [1mINFO [0m | [36mskyrl_train.utils.trainer_utils[0m:[36mbuild_dataloader[0m:[36m656[0m - [1mTotal steps: 156[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.322[0m | [1mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m357[0m - [1mLength of train_dataloader: 5000[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.322[0m | [1mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m358[0m - [1mNumber of steps per epoch: 78[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:54:37.322[0m | [1mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m359[0m - [1mTotal training steps: 80[0m
|
||
[33m(raylet, ip=10.128.25.27)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 100x across cluster][0m
|
||
[36m(bundle_reservation_check_func pid=364729)[0m [2026-06-12 00:54:42,131 E 364729 364769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[33m(raylet, ip=10.128.25.172)[0m 2026-06-12 00:54:38,341 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 16x across cluster][0m
|
||
[36m(pid=326289, ip=10.128.25.27)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 20x across cluster][0m
|
||
[36m(pid=326289, ip=10.128.25.27)[0m No module named 'vllm._version'[32m [repeated 20x across cluster][0m
|
||
[36m(pid=326289, ip=10.128.25.27)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 20x across cluster][0m
|
||
[36m(pid=441858, ip=10.128.25.172)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
|
||
[33m(raylet, ip=10.128.25.172)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 7x across cluster][0m
|
||
[36m(pid=316064, ip=10.128.25.23)[0m [2026-06-12 00:54:48,004 E 316064 316166] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 7x across cluster][0m
|
||
[36m(pid=300698, ip=10.128.25.162)[0m [2026-06-12 00:54:53,488 E 300698 300807] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster][0m
|
||
[36m(pid=443232, ip=10.128.25.174)[0m [2026-06-12 00:54:59,702 E 443232 443308] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 12x across cluster][0m
|
||
[36m(pid=305554, ip=10.128.25.164)[0m [2026-06-12 00:55:06,467 E 305554 305664] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:02.324 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:02.338 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:02.338 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
|
||
[36m(pid=441858, ip=10.128.25.172)[0m [2026-06-12 00:55:08,360 E 441858 441898] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:03.476 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:03.499 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:03.499 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:03.499 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:03.499 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=1.0, top_p=1.0, top_k=-1
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 00:57:03.546 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 2.14s (attempt 1/5) to avoid port collisions
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m No module named 'vllm._version'
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:03.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:03.393 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:167 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:03.394 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:170 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 2026-06-12 00:57:08 INFO [ipv4-debug] hostname=jpbo-029-44.jupiter.internal
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 2026-06-12 00:57:08 INFO [ipv4-debug] _global_node.node_ip_address=10.128.25.172
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 2026-06-12 00:57:08 INFO [ipv4-debug] get_node_ip_address()=10.128.25.172
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:04.715 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:04.731 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:04.731 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:618 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:04.731 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:626 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 2026-06-12 00:57:04.731 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1192 - OpenAI API sampling params overridden: temperature=1.0, top_p=1.0, top_k=-1[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m 2026-06-12 00:57:04.700 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1244 - Engine startup stagger: sleeping 1.80s (attempt 1/5) to avoid port collisions[32m [repeated 47x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 00:57:08.981[0m | [1m[32mINFO [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m636[0m - [1m[32mInitializing process group for RayActorGroup[0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [W612 00:57:09.772933846 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-44.jupiter.internal]:44913 (errno: 97 - Address family not supported by protocol).
|
||
[33m(raylet, ip=10.128.25.172)[0m 2026-06-12 00:57:09,981 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
[36m(pid=364947)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
|
||
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) _C._set_float32_matmul_precision(precision)
|
||
[33m(raylet)[0m 2026-06-12 00:57:10,083 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.[32m [repeated 6x across cluster][0m
|
||
[36m(pid=441955, ip=10.128.25.172)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m [W612 00:57:27.377459927 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-46-interconnect-1.jupiter.internal]:47029 (errno: 97 - Address family not supported by protocol).
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m [W612 00:57:27.379402447 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m [rank0]:[W612 00:57:27.384365575 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488)
|
||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:13, 4.60s/it]
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m [W612 00:57:27.641765305 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-46-interconnect-1.jupiter.internal]:56675 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m [W612 00:57:27.642268302 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m [rank0]:[W612 00:57:27.644246917 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
|
||
[36m(pid=441953, ip=10.128.25.172)[0m [2026-06-12 00:57:39,999 E 441953 442066] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494)
|
||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.33s/it][32m [repeated 7x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488)
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505)
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484)
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494)
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(pid=364945)[0m [2026-06-12 00:57:40,202 E 364945 365175] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00, 3.57s/it][32m [repeated 12x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m 2026-06-12 00:57:49.377 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m 2026-06-12 00:57:49.406 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m No module named 'vllm._version'
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m [W612 01:01:03.253829049 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-44.jupiter.internal]:44913 (errno: 97 - Address family not supported by protocol).
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m [W612 01:01:03.254567369 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 87x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 44x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 86x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m No module named 'vllm._version'[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 43x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:06.020[0m | [1m[32mINFO [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m638[0m - [1m[32mInitialized process group for RayActorGroup[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:06.030[0m | [1m[32mINFO [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m640[0m - [1m[32mMesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m `torch_dtype` is deprecated! Use `dtype` instead!
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m
|
||
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m
|
||
Loading checkpoint shards: 25%|██▌ | 1/4 [00:01<00:03, 1.22s/it]
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m [W612 01:01:05.915738414 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-44-interconnect-1.jupiter.internal]:44913 (errno: 97 - Address family not supported by protocol).[32m [repeated 6x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m [W612 01:01:05.005877978 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m `torch_dtype` is deprecated! Use `dtype` instead![32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m
|
||
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:04<00:01, 1.64s/it]
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.21s/it]
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.22s/it]
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m
|
||
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s][32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m
|
||
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:04<00:01, 1.66s/it][32m [repeated 19x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349991) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479189) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479185) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=326287, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326760) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326772) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317118) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317128) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316639, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317124) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161)[0m (EngineCore_DP0 pid=350003) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316471) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316475) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=478791, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479193) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301668) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301661) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301664) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326770) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=315936, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316481) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m [W612 01:01:16.718846230 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-026-44-interconnect-1.jupiter.internal]:51703 (errno: 97 - Address family not supported by protocol).
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m [W612 01:01:16.719483719 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m [rank0]:[W612 01:01:16.888041039 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479197) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m
|
||
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:04<00:01, 1.65s/it]
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.21s/it][32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301662) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317120) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=349500, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349997) _C._set_float32_matmul_precision(precision)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m
|
||
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.22s/it][32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326764) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316483) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349991)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=379729, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380138) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=379730, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380139) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305976) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=300569, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301120) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=300697, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301112) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249632) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249633) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249625) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380145) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301118) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305968) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249637) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380144) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301108) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301108) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479197)
|
||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:10, 3.44s/it]
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m [W612 01:01:19.363117061 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-34-interconnect-1.jupiter.internal]:35647 (errno: 97 - Address family not supported by protocol).[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m [W612 01:01:19.363562812 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m [rank0]:[W612 01:01:19.365486709 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301118)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999)
|
||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.23s/it][32m [repeated 60x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479197)
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479197)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00, 3.51s/it][32m [repeated 41x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479189)
|
||
[36m(AsyncVLLMInferenceEngine pid=478791, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479193)
|
||
[36m(AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479185)
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317118)
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317128)
|
||
[36m(AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317120)
|
||
[36m(AsyncVLLMInferenceEngine pid=316639, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317124)
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301668)
|
||
[36m(AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301661)
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301662)
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301664)
|
||
[36m(AsyncVLLMInferenceEngine pid=315936, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316481)
|
||
[36m(AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316471)
|
||
[36m(AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316483)
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316475)
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326770)
|
||
[36m(AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326764)
|
||
[36m(AsyncVLLMInferenceEngine pid=326287, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326760)
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m (EngineCore_DP0 pid=326772)
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999)
|
||
[36m(AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161)[0m (EngineCore_DP0 pid=350003)
|
||
[36m(AsyncVLLMInferenceEngine pid=349500, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349997)
|
||
[36m(AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349991)
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380145)
|
||
[36m(AsyncVLLMInferenceEngine pid=379729, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380138)
|
||
[36m(AsyncVLLMInferenceEngine pid=379730, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380139)
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380144)
|
||
[36m(AsyncVLLMInferenceEngine pid=300569, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301120)
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301108)
|
||
[36m(AsyncVLLMInferenceEngine pid=300697, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301112)
|
||
[36m(AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301118)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249632)
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249633)
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249625)
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249637)
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964)
|
||
[36m(AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305968)
|
||
[36m(AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305976)
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m [rank5]:[W612 01:01:35.443417320 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316471) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m 2026-06-12 01:01:37.057 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:15<00:00, 3.78s/it][32m [repeated 98x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:38.211[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mbuild_models[0m:[36m851[0m - [1m[32minit policy/ref/critic models done[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:38.216[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_maybe_enable_rollout_fanout[0m:[36m456[0m - [1m[32mRollout fan-out ENABLED: replacing single-process generator with RolloutDispatcher (K=4, cpus_per_coordinator=8).[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:38.239[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36m__init__[0m:[36m387[0m - [1m[32m[RolloutDispatcher] fan-out path: overriding inference host 127.0.0.1 -> 10.128.25.17 (routable head IP) for coordinator litellm base_url connectivity[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:38.239[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36m__init__[0m:[36m405[0m - [1m[32m[RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:38.264[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m434[0m - [1m[32m[RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD)[0m
|
||
[33m(raylet, ip=10.128.25.161)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[33m(raylet, ip=10.128.25.161)[0m 2026-06-12 01:01:39,065 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.319 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m [rank2]:[W612 01:01:35.573353744 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 7x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164)[0m 2026-06-12 01:01:39.969 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 39x across cluster][0m
|
||
[33m(raylet, ip=10.128.25.161)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 5x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:44.531[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 1/4 started[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.526 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.526 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.526 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.527 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.25.17:8000)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.530 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.530 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:01:44.530 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete
|
||
[33m(raylet, ip=10.128.25.176)[0m 2026-06-12 01:01:47,209 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.275 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[33m(raylet, ip=10.128.25.176)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:51.491[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 2/4 started[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.487 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.25.17:8000)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.490 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.490 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m 2026-06-12 01:01:51.490 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete
|
||
[33m(raylet, ip=10.128.25.162)[0m 2026-06-12 01:01:54,176 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.518 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[33m(raylet, ip=10.128.25.162)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.740 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.25.17:8000)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.743 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.743 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:01:58.743 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:01:58.744[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 3/4 started[0m
|
||
[33m(raylet, ip=10.128.25.25)[0m 2026-06-12 01:02:01,691 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.745 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
|
||
[33m(raylet, ip=10.128.25.25)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.967 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.25.17:8000)
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.970 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:298 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.971 | INFO | examples.terminal_bench.terminal_bench_generator:startup:255 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:02:07.971 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.972[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m469[0m - [1m[32m[RolloutDispatcher] coordinator 4/4 started[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.972[0m | [1m[32mINFO [0m | [36mexamples.terminal_bench.rollout_coordinator[0m:[36mstartup[0m:[36m477[0m - [1m[32m[RolloutDispatcher] 4 coordinators started[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.972[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36mtrain[0m:[36m485[0m - [1m[32mGenerator startup complete[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.972[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m510[0m - [1m[32mStarted: 'load_checkpoints'[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.975[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1735[0m - [1m[32mLoading checkpoint from: /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.975[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1741[0m - [1m[32mResuming from global_step: 82[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.981[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1757[0m - [1m[32mSuccessfully loaded trainer state[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.983[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1767[0m - [1m[32mSuccessfully loaded dataloader state[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:07.983[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1776[0m - [1m[32mLoading policy checkpoint from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [2026-06-12 01:02:09,083 E 350242 350282] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [2026-06-12 01:02:17,227 E 479438 479478] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [2026-06-12 01:02:24,194 E 301361 301401] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m warnings.warn( # warn only once
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.436[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1786[0m - [1m[32mSuccessfully loaded policy checkpoint[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.436[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mload_checkpoints[0m:[36m1802[0m - [1m[32mSuccessfully loaded complete checkpoint state from global_step_82[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.436[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m512[0m - [1m[32mResumed training from global_step 82[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.436[0m | [33m[1mWARNING [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m517[0m - [33m[1mNo data consumption state found in checkpoint — resume may re-train on already-consumed data[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.437[0m | [33m[1mWARNING [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m537[0m - [33m[1mData consumption count mismatch on resume: expected 256, got 0. This can happen after epoch boundary transitions or error recovery.[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.437[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m510[0m - [1m[32mFinished: 'load_checkpoints', time cost: 21.47s[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:02:29.437[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m544[0m - [1m[32mStarted: 'init_weight_sync_state'[0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m No module named 'vllm._version'
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m from .version import __version__, __version_tuple__ # isort:skip
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [2026-06-12 01:02:31,710 E 380389 380429] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [32m2026-06-12 01:02:35.472[0m | [1m[32mINFO [0m | [36mlogging[0m:[36minfo[0m:[36m2216[0m - [1m[32m[weight-sync] Using master_addr=10.128.25.172, master_port=42815[0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [32m2026-06-12 01:02:35.472[0m | [1m[32mINFO [0m | [36mlogging[0m:[36minfo[0m:[36m2216[0m - [1m[32m[weight-sync] get_node_ip_address()=10.128.25.172[0m[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m No module named 'vllm._version'[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m from .version import __version__, __version_tuple__ # isort:skip[32m [repeated 7x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304987) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304984) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) _C._set_float32_matmul_precision(precision)
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m [W612 01:05:00.280891184 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-37.jupiter.internal]:33847 (errno: 97 - Address family not supported by protocol).
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m [W612 01:05:00.281316200 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m [rank0]:[W612 01:05:00.286139823 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985)
|
||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:14, 4.85s/it]
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m [W612 01:05:00.280943535 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-37-interconnect-1.jupiter.internal]:34269 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m [W612 01:05:00.281413062 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m [rank0]:[W612 01:05:00.286345611 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986)
|
||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985)
|
||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:13<00:04, 4.48s/it][32m [repeated 8x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985)
|
||
[36m(AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304987)
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304984)
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986)
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986)
|
||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:15<00:00, 3.79s/it][32m [repeated 11x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m 2026-06-12 01:05:25.070 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380145) 2026-06-12 01:05:25.142 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:223 - torch.distributed.get_rank(): 0, rank_offset: 5, rank: 5, world_size: 49, group_name: skyrl
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m [rank0]:[W612 01:05:25.941600277 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-44-interconnect-1.jupiter.internal]:42815 (errno: 97 - Address family not supported by protocol).
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380145) 2026-06-12 01:05:25.145 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:234 - init_weight_update_communicator: master_address=10.128.25.172, master_port=42815,
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m warnings.warn( # warn only once
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:05:25.199[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36minit_weight_sync_state[0m:[36m874[0m - [1m[32mInitialized weight sync state for policy model and inference engines.[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:05:25.200[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m544[0m - [1m[32mFinished: 'init_weight_sync_state', time cost: 175.76s[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:05:25.200[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m548[0m - [1m[32mStarted: 'sync_weights_to_inference_engines'[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m [rank0]:[W612 01:05:29.963020285 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:05:34.535[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m548[0m - [1m[32mFinished: 'sync_weights_to_inference_engines', time cost: 9.34s[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:05:34.536[0m | [1m[32mINFO [0m | [36mskyrl_train.callbacks.builtin[0m:[36mon_train_begin[0m:[36m270[0m - [1m[32mHFHubUploadCallback initialized: repo=laion/explore-tis-untrunc, upload_steps=5, export_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m
|
||
Training Step Progress: 82it [00:00, ?it/s]
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m 2026-06-12 01:05:25.070 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1301 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249637) 2026-06-12 01:05:25.145 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:223 - torch.distributed.get_rank(): 0, rank_offset: 24, rank: 24, world_size: 49, group_name: skyrl[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m [rank0]:[W612 01:05:25.561917791 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-029-44.jupiter.internal]:42815 (errno: 97 - Address family not supported by protocol).[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m (EngineCore_DP0 pid=249637) 2026-06-12 01:05:25.148 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:init_weight_update_communicator:234 - init_weight_update_communicator: master_address=10.128.25.172, master_port=42815, [32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m [rank0]:[W612 01:05:29.579234427 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 47x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [32m2026-06-12 01:08:53.812[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m604[0m - [1m[32mStarted: 'step'[0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m
|
||
Generation Buffer Progress: 0%| | 0/64 [00:00<?, ?it/s][A[32m2026-06-12 01:08:53.812[0m | [1m[32mINFO [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_train_loop[0m:[36m607[0m - [1m[32mStarted: 'wait_for_generation_buffer'[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m 2026-06-12 01:08:53.816 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m 2026-06-12 01:09:06.223 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)[32m [repeated 119x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m 2026-06-12 01:09:08.719 | INFO | examples.terminal_bench.terminal_bench_generator:generate:560 - Starting batch generation for 8 trials (mode=training)[32m [repeated 218x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 95x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379729, ip=10.128.25.25)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 28x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 13x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 10x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 13x across cluster][0m
|
||
slurmstepd: error: *** JOB 736687 ON jpbo-026-33 CANCELLED AT 2026-06-12T01:10:56 ***
|
||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/ray_logs/
|
||
2026-06-12 01:10:57.000 | ERROR | __main__:main:134 - Training failed: Task failed because the node it was running on is dead or unavailable. Node IP: 10.128.25.17, node ID: 96706d0fc7d05b397467667c5e336ac21712cf7e0c8bf1b2f4037765. This can happen if the node was preempted, had a hardware failure, or its raylet crashed unexpectedly. To see node death information, use `ray list nodes --filter node_id=96706d0fc7d05b397467667c5e336ac21712cf7e0c8bf1b2f4037765`, check the Ray dashboard cluster page, search the node ID in the GCS logs, or use `ray logs raylet.out -ip 10.128.25.17`.
|
||
2026-06-12 01:10:57.000 | INFO | __main__:main:137 - Shutting down Ray on head node...
|
||
[36m(skyrl_entrypoint pid=364638)[0m /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown
|
||
[36m(skyrl_entrypoint pid=364638)[0m warnings.warn('resource_tracker: There appear to be %d '
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 2x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [00:54:01] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [00:54:01] OK: RSS 1.38 GiB | node mem 135.5/858.0 GiB used (15.8%), avail 722.5 GiB
|
||
[36m(skyrl_entrypoint pid=364638)[0m ⚙️ Running in WANDB offline mode
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [00:56:01] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [00:56:01] OK: RSS 1.64 GiB | node mem 136.1/858.0 GiB used (15.9%), avail 721.9 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m WARNING 06-12 00:57:06 [arg_utils.py:1256] The global random seed is set to 69. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m INFO 06-12 00:57:06 [model.py:529] Resolved architecture: Qwen3ForCausalLM
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m INFO 06-12 00:57:06 [model.py:1549] Using max model len 32768
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m INFO 06-12 00:57:06 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m INFO 06-12 00:57:06 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m INFO 06-12 00:57:06 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m WARNING 06-12 00:57:06 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m INFO 06-12 00:57:06 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m WARNING 06-12 00:57:06 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m WARNING 06-12 00:57:06 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
|
||
[36m(pid=441858, ip=10.128.25.172)[0m ⚙️ Running in WANDB offline mode
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:19 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=67, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m WARNING 06-12 00:57:07 [arg_utils.py:1256] The global random seed is set to 68. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 00:57:07 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 00:57:07 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 00:57:07 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 00:57:07 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 00:57:07 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m WARNING 06-12 00:57:07 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 00:57:07 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m WARNING 06-12 00:57:07 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m WARNING 06-12 00:57:07 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) INFO 06-12 00:57:26 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:19 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=78, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:27 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.174:47029 backend=nccl
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:27 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:29 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:44 [default_loader.py:293] Loading weights took 14.39 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:27 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:27 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.174:56675 backend=nccl[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:27 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:29 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:44 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.864247 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443505) INFO 06-12 00:57:48 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:48 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.25 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) WARNING 06-12 00:57:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:49 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) WARNING 06-12 00:57:49 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:49 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:44 [default_loader.py:293] Loading weights took 14.27 seconds[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m WARNING 06-12 00:57:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [00:58:01] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [00:58:01] OK: RSS 1.64 GiB | node mem 137.6/858.0 GiB used (16.0%), avail 720.3 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:44 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.860758 seconds[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:48 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:48 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443484) INFO 06-12 00:57:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443488) INFO 06-12 00:57:48 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:48 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.22 seconds[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) WARNING 06-12 00:57:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:49 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) WARNING 06-12 00:57:49 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m (EngineCore_DP0 pid=443494) INFO 06-12 00:57:49 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m WARNING 06-12 00:57:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [01:00:01] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [01:00:01] OK: RSS 1.64 GiB | node mem 137.7/858.0 GiB used (16.0%), avail 720.3 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m WARNING 06-12 01:00:55 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m INFO 06-12 01:00:55 [model.py:529] Resolved architecture: Qwen3ForCausalLM
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m INFO 06-12 01:00:55 [model.py:1549] Using max model len 32768
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m INFO 06-12 01:00:55 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m INFO 06-12 01:00:55 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m INFO 06-12 01:00:55 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m WARNING 06-12 01:00:55 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m INFO 06-12 01:00:55 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176)[0m WARNING 06-12 01:00:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163)[0m WARNING 06-12 01:00:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
|
||
[36m(pid=364947)[0m ⚙️ Running in WANDB offline mode
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m WARNING 06-12 01:00:55 [arg_utils.py:1256] The global random seed is set to 65. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m INFO 06-12 01:00:55 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m INFO 06-12 01:00:55 [model.py:1549] Using max model len 32768[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m INFO 06-12 01:00:55 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_DEBUG_SUBSYS': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'RAY_USE_UVLOOP': '***', 'SKYRL_WORKER_NCCL_TIMEOUT_IN_S': '***', 'TORCH_FR_BUFFER_SIZE': '***', 'TORCH_NCCL_ASYNC_ERROR_HANDLING': '***', 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': '***', 'TORCH_NCCL_DUMP_ON_TIMEOUT': '***', 'TORCH_NCCL_TRACE_BUFFER_SIZE': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m INFO 06-12 01:00:55 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m INFO 06-12 01:00:55 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m WARNING 06-12 01:00:55 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m INFO 06-12 01:00:55 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m WARNING 06-12 01:00:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m WARNING 06-12 01:00:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301668) INFO 06-12 01:01:09 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=66, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
|
||
[36m(pid=441955, ip=10.128.25.172)[0m ⚙️ Running in WANDB offline mode[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999) INFO 06-12 01:01:15 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380144) INFO 06-12 01:01:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=49, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 43x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999) INFO 06-12 01:01:15 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.161:54467 backend=nccl
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999) INFO 06-12 01:01:16 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349999) INFO 06-12 01:01:16 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
|
||
[36m(AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161)[0m (EngineCore_DP0 pid=349991) INFO 06-12 01:01:17 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301108) INFO 06-12 01:01:19 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479197) INFO 06-12 01:01:32 [default_loader.py:293] Loading weights took 14.07 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301108) INFO 06-12 01:01:19 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.162:35647 backend=nccl[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380144) INFO 06-12 01:01:19 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m (EngineCore_DP0 pid=380144) INFO 06-12 01:01:19 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301118) INFO 06-12 01:01:20 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479197) INFO 06-12 01:01:32 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.462349 seconds
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO cudaDriverVersion 13020
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.172<0>
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Initialized NET plugin IB
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Assigned NET plugin IB to comm
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Using network IB
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO DMA-BUF is available on GPU device 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO ncclCommInitRankConfig comm 0x400dfa6b40c0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 901000 commId 0xd750aa8fd3370a86 - Init START
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Setting affinity for GPU 0 to 0-71
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO comm 0x400dfa6b40c0 rank 0 nRanks 8 nNodes 2 localRanks 4 localRank 0 MNNVL 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/08 : 0 1 2 3 7 6 5 4
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 01/08 : 0 4 5 6 7 3 2 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/08 : 0 3 1 5 4 7 6 2
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 03/08 : 0 3 2 6 4 7 5 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/08 : 0 1 2 3 7 6 5 4
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 05/08 : 0 4 5 6 7 3 2 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/08 : 0 3 1 5 4 7 6 2
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 07/08 : 0 3 2 6 4 7 5 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Trees [0] 1/4/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->3 [4] 1/-1/-1->0->4 [5] -1/-1/-1->0->3 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->3
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO P2P Chunksize set to 131072
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO [Proxy Service] Device 0 CPU core 49
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442377 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c000000 size 2097152 ipcDesc 0x400ed4008b70
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c200000 size 2097152 ipcDesc 0x400ed4009b30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c400000 size 2097152 ipcDesc 0x400ed400a940
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 03/0 : 0[0] -> 3[3] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c600000 size 2097152 ipcDesc 0x400ed400b730
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118c800000 size 2097152 ipcDesc 0x400ed400c500
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 07/0 : 0[0] -> 3[3] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118ca00000 size 2097152 ipcDesc 0x400ed400d2d0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442383 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 51
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/0 : 4[0] -> 0[0] [receive] via NET/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/0 : 4[0] -> 0[0] [receive] via NET/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 01/0 : 0[0] -> 4[0] [send] via NET/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 05/0 : 0[0] -> 4[0] [send] via NET/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118cc00000 size 10485760 ipcDesc 0x400ed4010180
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118d600000 size 10485760 ipcDesc 0x400ed4010f50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118e000000 size 10485760 ipcDesc 0x400ed4010f50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118ea00000 size 10485760 ipcDesc 0x400ed4010f50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118f400000 size 10485760 ipcDesc 0x400ed4010f50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x40118fe00000 size 10485760 ipcDesc 0x400ed4010f50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401190800000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401191200000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401191c00000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401192600000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401193000000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401193a00000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194400000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194600000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194800000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194a00000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194c00000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401194e00000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195000000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195200000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195400000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195600000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195800000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 07/0 : 0[0] -> 2[2] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195a00000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195c00000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/IPC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401195e00000 size 2097152 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 00/0 : 0[0] -> 4[0] [send] via NET/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO Channel 04/0 : 0[0] -> 4[0] [send] via NET/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401196000000 size 10485760 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0] NCCL INFO Allocated shareable buffer 0x401196a00000 size 10485760 ipcDesc 0x400ed4015c30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442374 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442347 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442376 [0] NCCL INFO Allocated
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365456 [0] NCCL INFO Connected all trees
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365456 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365456 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365456 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365456 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e412e5990 rank 5 nranks 8 cudaDev 0 nvmlDev 1 busId 1901000 commId 0xd750aa8fd3370a86 - Init COMPLETE
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365456 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 5 nranks 8 total 2.83 (kernels 0.32, alloc 1.84, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.63, rest 0.00)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442346 [0] NCCL INFO CC Off, workFifoBytes 1048576
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 0 sendbuff 0x401340000000 recvbuff 0x401340000000 count 622329856 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 00/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 01/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 02/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 03/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 04/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 05/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 06/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 07/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 08/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 09/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 10/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 11/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 12/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 13/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 14/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 15/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 16/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 17/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 18/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 19/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 20/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 21/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 22/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 23/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 24/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 25/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 26/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 27/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 28/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 29/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 30/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 31/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 32/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 33/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 34/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 35/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 36/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 37/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 38/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 39/40 : 0 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[0] [send] via NE
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442389 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[1] [send]
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m e400000
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442391 [0] NCCL INFO Channel 01/0 : 0[3] -> 1[3] [se
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m shareable buffer 0x400f1fc00000 size 2097152 ipcDesc 0x400f6429ffb0
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301662) INFO 06-12 01:01:36 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301662) INFO 06-12 01:01:36 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m (EngineCore_DP0 pid=301662) INFO 06-12 01:01:36 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m (EngineCore_DP0 pid=479189) INFO 06-12 01:01:36 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m T/IB/0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO Connected binomial trees
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442393 [0] NCCL INFO CC Off, workFifoBytes 1048576
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m via NET/IB/1
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m nd] via NET/IB/3
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317118) INFO 06-12 01:01:36 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.74 seconds
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jp
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m fc00000
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ed30
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 400ff000ed30
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Imported shareable buffer device 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442407 [0] NCCL INFO ncclCommInitRankConfig comm 0x400dfaa9fd00 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 901000 commId 0x8d21f48cdf4c109b - Ini
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:44
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m ff800ed30
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442412 [0] NCCL INFO Alloc
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m 0
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:4419
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400f40000000 count 155582464 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365517 [
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m IPC
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013528c0000 count 155582464 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401365180000 count 155582464 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401377a40000 count 155582464 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401100800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101000000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101800000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401116a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401116c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401116e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4011c3a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4011c3c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4011c3e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401100800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101000000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401101800000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa02ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401135400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401138400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount a sendbuff 0x400edfa00000 recvbuff 0x400edfa00000 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00800 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01000 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01800 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount b sendbuff 0x400edfa31400 recvbuff 0x400edfa31400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa31c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401135c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401103000000 recvbuff 0x401103000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103200000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103400000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103600000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount a sendbuff 0x401103800000 recvbuff 0x401103800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401103800000 recvbuff 0x401103800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401103e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount b sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401135c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401136c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x400edfa03e00 recvbuff 0x400edfa03e00 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa03e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa03e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa03ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x400edfa04200 recvbuff 0x400edfa04200 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa042c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount d sendbuff 0x40113b400000 recvbuff 0x40113b400000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110a800000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40110c000000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x400edfa00000 recvbuff 0x400edfa00000 count 4096 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00800 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01000 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01800 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x400edfa31c00 recvbuff 0x400edfa31c00 count 4096 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401107800000 recvbuff 0x401107800000 count 16777216 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401107c00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401107e00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108000000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401108c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401109c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa04280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa042c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa00ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113bc00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113c400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113d600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113d800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113da00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113b600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113b800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113ba00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113c400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113cc00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40113d400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401347800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134a800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341000000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341800000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401140e00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401141000000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401141200000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401340800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341200000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401341a00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401342200000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa01ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa32680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa326c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401342200000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343a00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401345200000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349a00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ca00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401352a00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344200000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344a00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401345200000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401346400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401346600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401346800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343c00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401343e00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401344000000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401348a00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349200000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349a00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa336c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401349a00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ca00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401352a00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401355a00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357200000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401358a00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ba00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134c200000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ca00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b400000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b600000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134b800000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134c400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134cc00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa33ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa34ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134d400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134ec00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135ac00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135c400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135dc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fc00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401351a00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401353c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa35ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa362c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401354c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135c400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135dc00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401365400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ac00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401358e00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401359000000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401359200000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401356c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401357c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa36280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa362c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa37680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa376c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135c400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135dc00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401365400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136b400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ac00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3bc00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135e400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135ec00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134f800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fa00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40134fc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401350400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135e400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135ec00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa386c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40135f400000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401360c00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401366c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3a400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ac00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361c00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362e00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363000000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363200000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363a00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361400000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401361c00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401362400000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa38ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39e40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39e80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa39ec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401366c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136cc00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136e400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136fc00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372c00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374400000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375c00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3e400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368c00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369400000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369c00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363600000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363800000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401363a00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368a00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401368c00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401369e00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136a600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ae40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ae80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3aec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b2c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136a600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136be00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136d600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401377e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401379600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ae00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3cc00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136ce00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136d600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136e800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136ea00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136ec00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40136c400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401370e00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371e00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3b2c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3c6c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401371e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401377e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365515 [0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m (nil) recvbuff 0x401379600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ae00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137de00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401380e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384e00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401385600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401384400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401385000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401385800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401386000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3d6c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3dac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401379600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ae00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137c600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374e00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ec00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3f400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fc00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401389a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401389c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401389e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3da80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3dac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ee40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3ee80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3eec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137b600000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ce00000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137e600000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137be00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137c600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ce00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401381800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401381a00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401381c00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137b800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ba00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137bc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137c600000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ce00000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137d600000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x400edfa3fe00 recvbuff 0x400edfa3fe00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fe40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fe80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa3fec0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x400edfa40200 recvbuff 0x400edfa40200 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa402c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x401372600000 recvbuff 0x401372600000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x40164f000000 recvbuff 0x40164f000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401650800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401652000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401653800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x401372600000 recvbuff 0x401372600000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount af sendbuff 0x400edfa40c00 recvbuff 0x400edfa40c00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41400 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41c00 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42400 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edf
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m a44c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401652000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f000000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f200000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f400000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137f800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137fa00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137fc00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651000000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401651800000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401652000000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa40280 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa402c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa41680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa416c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401656800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401658000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401659800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401375600000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401376e00000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x400edfa43400 recvbuff 0x400edfa43400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa43c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa44c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x400edfa45400 recvbuff 0x400edfa45400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa45c00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa46400 count 1024 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa46c00 count 1024 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x401372600000 recvbuff 0x401372600000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373600000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373e00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x401374600000 recvbuff 0x401374600000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374800000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374a00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374c00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount be sendbuff 0x401372800000 recvbuff 0x401372800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372a00000 count 1048576 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372c00000 count 1048576 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401372e00000 count 1048576 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x401373200000 recvbuff 0x401373200000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401373a00000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374200000 count 4194304 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401374a00000 count 4194304 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42640 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42680 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa426c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42a40 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42a80 count 32 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa42ac0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401656800000 count 12582912 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401658000000 count 12582912 datatype 9 op 0 root 2 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401659800000 count 12582912 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:3
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m t COMPLETE
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Broadcast: opCount 8b sendbuff 0x4010910000
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x401605800000 recvbuff 0x401605800000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x400edfa42a00 recvbuff 0x400edfa42a00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opC
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m ated shareable buffer 0x32e600000 size 2097152 ipcDesc 0x400ff800ed30
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO Broadcast: opCount 90 sendbuff 0x401091000000 recvbuff 0x401091000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e255
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m 00 recvbuff 0x401091000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e115b17e0 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Broadcast: opCount 63 sendbuff 0x400e99e13e00 recvbuff 0x400e99e13e00 count 4096 datatyp
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m af260 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:4419
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount 63 sendbuff 0x400e99e13e00 recvbuff 0x400e99e13e00 count 4096 datatype
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m ount 83 sendbuff 0x400edfa43e00 recvbuff 0x400edfa43e00 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x400edfa43e00 recvbuff 0x400edfa43e00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x401655000000 recvbuff 0x401655000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x40160b000000 recvbuff 0x40160b000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x401655000000 recvbuff 0x401655000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount da sendbuff 0x401655000000 recvbuff 0x401655000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount db sendbuff 0x400edfa45c00 recvbuff 0x400edfa45c00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x400edfa48400 recvbuff 0x400edfa48400 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x401611000000 recvbuff 0x401611000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401613000000 recvbuff 0x401613000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401611000000 recvbuff 0x401611000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401617000000 recvbuff 0x401617000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x401617800000 recvbuff 0x401617800000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x400edfa47600 recvbuff 0x400edfa47600 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x400edfa47a00 recvbuff 0x400edfa47a00 count 128 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401617000000 recvbuff 0x401617000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount f9 sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 5033164
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m size 10485760 ptr 0x334400000
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 8 sendbuff 0x400e99e02e00 recvbuff 0x400e99e02e00 count 4096 datatype 9 op 0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m 8 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x4013e6000000 recvbuff 0x4013e6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x400edfa4ac00 recvbuff 0x400edfa4ac00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400edfa4cc00 recvbuff 0x400edfa4cc00 count 4096 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x40161b800000 recvbuff 0x40161b800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount 102 sendbuff 0x400edfa49e00 recvbuff 0x400edfa49e00 count 128 datatype 9 op 0 ro
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ] NCCL INFO Allocated shareable buffer 0x401024600000 size 10485760 ipcDesc 0x400fd000ed30
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount 8 sendbuff 0x400e99e02e00 recvbuff 0x400e99e02e00 count 4096 datatype 9 op 0 r
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Br
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m root 0 comm 0x400e412e5990 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:36
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m oot 0 comm 0x400e412e4700 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m oadcast: opCount 10 sendbuff 0x400eb9e00e00 recvbuff 0x400eb9e00e00 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 1f sendbuff 0x400ed9000000 recvbuff 0x400ed
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400ed0000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e6
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 37 sendbuff 0x400e99e07e00 recvbuff 0x400e99e07e00 coun
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpb
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m t 4096 datatype 9 op 0 root 0 comm 0x400e415b09a0 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL I
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m ot 0 comm 0x400e06983940 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x4013ea800000 recvbuff 0x4013ea800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x4013eb000000 recvbuff 0x4013eb000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount 10b sendbuff 0x4013eb000000 recvbuff 0x4013eb000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06983940 [nranks=2] stream (
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL IN
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m NFO Broadcast: opCount 37 sendbuff 0x400ed0c00000 recvbuff 0x400ed0c00000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) rec
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m FO Broadcast: opCount 37 sendbuff 0x400ed0c00000 recvbuff 0x400ed0c00000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recv
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 4b sendbuff 0x400eef400000 recvbuff 0x400eef400000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e0b000 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nra
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m vbuff 0x400e99e0a600 count 1024 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m buff 0x400e99e0a600 count 1024 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount 53 sendbuff 0x401071000000 recvbuff 0x401071000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:36
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 53 sendbuff 0x401071000000 recvbuff 0x401071000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] strea
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast:
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m m (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m opCount 5c sendbuff 0x401071000000 recvbuff 0x401071000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recv
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m : opCount 5c sendbuff 0x401071000000 recvbuff 0x401071000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) rec
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x400edfa4a200 recvbuff 0x400edfa4a200 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x400edfa4b600 recvbuff 0x400edfa4b600 count 128 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x4013f2000000 recvbuff 0x4013f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Broadcast: opCount af sendbuff 0x4013e0000000 recvbuff 0x4013e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send:
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m buff 0x40107c000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401036c00000 recvbuff 0x401036c00000 count 50331648 datatype 9 op 0
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x401074000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m vbuff 0x40107c000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-0
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m ] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadc
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m opCount 0 sendbuff (nil) recvbuff 0x4013e0800000 count 4194304 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) re
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadca
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount c1 sendbuff 0x400eb9e15200 rec
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m ast: opCount 83 sendbuff 0x400e99e16200 recvbuff 0x400e99e16200 count 128 datatype 9 op 0 root 0 comm 0x400e412e5990 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400f80000000 c
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m st: opCount 83 sendbuff 0x400e99e16200 recvbuff 0x400e99e16200 count 128 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount d0 sendbuff 0x400eb9e16e00 recvbuff 0x400eb9e16e00 count 4096 datatype 9 o
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m e 9 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m ount 12582912 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m p 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcas
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-02
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m cvbuff 0x400edfa4dc00 count 1024 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013ff800000 count 12582912 datatype 9 op 0
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x400eb9e1be00 recvbuf
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400fa2
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m f 0x400eb9e1be00 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e1ee00 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400edfa4f240 count 32 datatype 9 op 0 root 1 comm 0x400e06a9e080 [nranks=4] stream (n
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:36494
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:3649
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL IN
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL I
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m FO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e25400 count 32 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m INFO Broadcast: opCount 146 sendbuff 0x4010b2000000 recvbuff 0x4010b2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] N
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m NFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e25400 count 32 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m CCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e27e00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCou
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m il)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-3
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m nt 163 sendbuff 0x4010b2000000 recvbuff 0x4010b2000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 172 sendbuff 0x4010e0000000 recvbuff
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e415b1b60 [nranks=2] st
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e415b09a0 [nranks=2] s
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m t: opCount 26 sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010efa00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send:
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m opCount 0 sendbuff (nil) recvbuff 0x400edfa542c0 count 32 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401367c00000 count 12582912 datatype 9 op 0 r
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317128) WARNING 06-12 01:01:36 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317128) INFO 06-12 01:01:36 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317128) WARNING 06-12 01:01:36 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m (EngineCore_DP0 pid=317128) INFO 06-12 01:01:36 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x40137ac00000 count 4194304 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nra
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m WARNING 06-12 01:01:37 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) r
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m ecvbuff 0x4015d2200000 count 4194304 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:35 [default_loader.py:293] Loading weights took 15.14 seconds[32m [repeated 39x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013c5800000 count 1
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount e8 sendbuff 0x401626000000 recvbuf
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:36 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 16.330888 seconds[32m [repeated 39x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:01:44] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:01:44] OK: RSS 0.78 GiB | node mem 372.6/858.0 GiB used (43.4%), avail 485.4 GiB
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO cudaDriverVersion 13020[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO NCCL version 2.27.7+cuda13.0[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365455 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. [32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365455 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.17<0>[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365455 [0] NCCL INFO Initialized NET plugin IB[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Assigned NET plugin IB to comm[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Using network IB[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO DMA-BUF is available on GPU device 0[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e616c3ee0 rank 3 nranks 4 cudaDev 0 nvmlDev 3 busId 3901000 commId 0xa71b72472f667c3e - Init START[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365455 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365455 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Setting affinity for GPU 3 to 216-287[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO comm 0x400e616c3ee0 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365511 [0] NCCL INFO Channel 23/24 : 0 3 2 1[32m [repeated 168x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2 [2] -1/-1/-1->3->2 [3] -1/-1/-1->3->2 [4] -1/-1/-1->3->1 [5] -1/-1/-1->3->1 [6] -1/-1/-1->3->1 [7] -1/-1/-1->3->1 [8] 2/-1/-1->3->0 [9] 2/-1/-1->3->0 [10] 2/-1/-1->3->0 [11] 2/-1/-1->3->0 [12] -1/-1/-1->3->2 [13] -1/-1/-1->3->2 [14] -1/-1/-1->3->2 [15] -1/-1/-1->3->2 [16] -1/-1/-1->3->1 [17] -1/-1/-1->3->1 [18] -1/-1/-1->3->1 [19] -1/-1/-1->3->1 [20] 2/-1/-1->3->0 [21] 2/-1/-1->3->0 [22] 2/-1/-1->3->0 [23] 2/-1/-1->3->0[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO P2P Chunksize set to 524288[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365455 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. [32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365511 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0[32m [repeated 5x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365516 [0] NCCL INFO [Proxy Service] Device 0 CPU core 274[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365519 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 276[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442410 [0] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC[32m [repeated 586x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365515 [0] NCCL INFO Allocated shareable buffer 0x401028200000 size 10485760 ipcDesc 0x400fd000ed30[32m [repeated 1175x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365523 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 216[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365494 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[3] [receive] via NET/IB/3[32m [repeated 346x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365494 [0] NCCL INFO Channel 39/0 : 1[3] -> 0[3] [send] via NET/IB/3[32m [repeated 340x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365529 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x40102aa00000[32m [repeated 1187x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442408 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Connected all trees[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 16 p2p channels per peer[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442349 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e616c3ee0 rank 3 nranks 4 cudaDev 0 nvmlDev 3 busId 3901000 commId 0xa71b72472f667c3e - Init COMPLETE[32m [repeated 22x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365512 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 3 nranks 4 total 0.17 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.14, rest 0.00)[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365511 [0] NCCL INFO CC Off, workFifoBytes 1048576[32m [repeated 5x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 186 sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)[32m [repeated 5447x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442390 [0] NCCL INFO Channel 01/0 : 0[2] -> 1[2] [se
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:38 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:38 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:38 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 39x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:39 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 39x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365494 [0] NCCL INFO Connected binomial trees[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m nd] via NET/IB/2
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305972) INFO 06-12 01:01:39 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.08 seconds[32m [repeated 39x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jp
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010d8200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=4] stream (nil)[32m [repeated 2329x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02ac0 count 32 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)[32m [repeated 1758x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount a sendbuff 0x401103000000 recvbuff 0x401103000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount b sendbuff 0x401135400000 recvbuff 0x401135400000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401107800000 recvbuff 0x401107800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401107800000 recvbuff 0x401107800000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m (nil) recvbuff 0x400eae800000 count 12582912 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x4013a0000000 recvbuff 0x4013a0000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x4013a0000000 recvbuff 0x4013a0000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x4013a0000000 recvbuff 0x4013a0000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x4013a6000000 recvbuff 0x4013a6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 22x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x4013a6000000 recvbuff 0x4013a6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount af sendbuff 0x4013a6000000 recvbuff 0x4013a6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4013af000
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)[32m [repeated 5x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount be sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x4013ac000000 recvbuff 0x4013ac000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount 15e sendbuff 0x401[32m [repeated 4x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 128 sendbuff 0x400eb9e20e00 recvbuff 0x400eb9e20e00 count 4096 datatype 9 op 0 root 0 comm 0x400e615a[32m [repeated 8x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m recvbuff 0x40134c400000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)[32m [repeated 5x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m nks=4] stream (nil)[32m [repeated 14x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 95 sendbuff 0x400eb9e10200 recvbuff 0x400eb9e10200 count 128 datatype
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x4013b5000000 recvbuff 0x4013b5000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x4013b5c00000 recvbuff 0x4013b5c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount da sendbuff 0x401620000000 recvbuff 0x401620000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount db sendbuff 0x4013c6800000 recvbuff 0x4013c6800000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x4013c6800000 recvbuff 0x4013c6800000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x401620400000 recvbuff 0x401620400000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401626000000 recvbuff 0x401626000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401626000000 recvbuff 0x401626000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401626000000 recvbuff 0x401626000000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x401628800000 recvbuff 0x401628800000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x401630800000 recvbuff 0x401630800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x401630800000 recvbuff 0x401630800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401630800000 recvbuff 0x401630800000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x401630800000 recvbuff 0x401630800000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 27x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount e9 sendbuff 0x4010a4000000 recvbuff 0x4010a4000000 count 4194[32m [repeated 6x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 2582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)[32m [repeated 20x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x4015d1a00000 recvbuff 0x4015d1a00000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)[32m [repeated 12x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x400e9fa4b400 recvbuff 0x400e9fa4b400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)[32m [repeated 10x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa4b400 recvbuff 0x400e9fa4b400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)[32m [repeated 9x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4015d1a00000 recvbuff 0x4015d1a00000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)[32m [repeated 9x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x4015d1a00000 recvbuff 0x4015d1a00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)[32m [repeated 9x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m root 1 comm 0x400e06a9e080 [nranks=4] stream (nil)[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m oot 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2a400 count 32 datatype 9 op 0 root 0 comm 0x400e41[32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount a8
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m sendbuff 0x4010a0000000 recvbuff 0x4010a0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m vbuff 0x400eb9e15200 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401034e00000 recvbuff 0x401034e00000 count 50331648 datatype 9 op 0
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2c
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m f 0x401626000000 count 16777216 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL [32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964) WARNING 06-12 01:01:39 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964) INFO 06-12 01:01:39 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964) WARNING 06-12 01:01:39 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305427, ip=10.128.25.164)[0m (EngineCore_DP0 pid=305964) INFO 06-12 01:01:39 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 39x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164)[0m WARNING 06-12 01:01:39 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 39x across cluster][0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:01:51] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:01:51] OK: RSS 0.78 GiB | node mem 359.9/858.0 GiB used (41.9%), avail 498.1 GiB
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:01:58] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:01:58] OK: RSS 0.78 GiB | node mem 371.0/858.0 GiB used (43.2%), avail 486.9 GiB
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] Started monitoring (every 120s)
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:02:07] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)[32m [repeated 2x across cluster][0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:02:07] OK: RSS 0.78 GiB | node mem 381.1/858.0 GiB used (44.4%), avail 476.9 GiB[32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy/model_world_size_8_rank_0.pt
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy/extra_state_world_size_8_rank_0.pt
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy/optim_world_size_8_rank_0.pt
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Successfully loaded model state dict
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Successfully loaded optimizer state
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Successfully loaded scheduler state
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints/global_step_82/policy
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m e00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 181 sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 181 sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2ae00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 182 sendbuff 0x4010ec000000 recvbuff 0x4010ec000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 182 sendbuff 0x4010ec000000 recvbuff 0x4010ec000000 count 16777216 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010e5000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 183 sendbuff 0x4010e5800000 recvbuff 0x4010e5800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 183 sendbuff 0x4010e5800000 recvbuff 0x4010e5800000 count 4194304 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010ec000000 count 1048576 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 184 sendbuff 0x4010ec200000 recvbuff 0x4010ec200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 184 sendbuff 0x4010ec200000 recvbuff 0x4010ec200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010e5800000 count 1048576 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 185 sendbuff 0x4010eca00000 recvbuff 0x4010eca00000 count 16777216 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 185 sendbuff 0x4010eca00000 recvbuff 0x4010eca00000 count 16777216 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010ec200000 count 4194304 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 186 sendbuff 0x400eb9e2b600 recvbuff 0x400eb9e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 186 sendbuff 0x400eb9e2b600 recvbuff 0x400eb9e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2b800 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 187 sendbuff 0x400eb9e2ba00 recvbuff 0x400eb9e2ba00 count 128 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 187 sendbuff 0x400eb9e2ba00 recvbuff 0x400eb9e2ba00 count 128 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2b600 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 188 sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 188 sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010eca00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 189 sendbuff 0x4010f8000000 recvbuff 0x4010f8000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 189 sendbuff 0x4010f8000000 recvbuff 0x4010f8000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010ee200000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x4010f2000000 recvbuff 0x4010f2000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010efa00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18b sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18b sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2ba00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18c sendbuff 0x400eb9e2f600 recvbuff 0x400eb9e2f600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18c sendbuff 0x400eb9e2f600 recvbuff 0x400eb9e2f600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2c200 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18d sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18d sendbuff 0x400eb9e2d600 recvbuff 0x400eb9e2d600 count 4096 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400eb9e2f600 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18e sendbuff 0x401100000000 recvbuff 0x401100000000 count 622329856 datatype 9 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Broadcast: opCount 18e sendbuff 0x401100000000 recvbuff 0x401100000000 count 622329856 datatype 9 op 0 root 0 comm 0x400e615a9820 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x40114a400000 count 155582464 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:03:44] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:03:44] OK: RSS 0.89 GiB | node mem 372.8/858.0 GiB used (43.4%), avail 485.2 GiB
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:03:51] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:03:51] OK: RSS 0.89 GiB | node mem 360.2/858.0 GiB used (42.0%), avail 497.8 GiB
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:03:58] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:03:58] OK: RSS 0.89 GiB | node mem 371.2/858.0 GiB used (43.3%), avail 486.8 GiB
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:04:07] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)[32m [repeated 2x across cluster][0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:04:07] OK: RSS 0.89 GiB | node mem 381.3/858.0 GiB used (44.4%), avail 476.7 GiB[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:04:59 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:00 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.165:33847 backend=nccl
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:00 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:19 [default_loader.py:293] Loading weights took 15.15 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:04:59 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:00 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.25.165:34269 backend=nccl[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:00 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:20 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.668826 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:23 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:23 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:23 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304984) INFO 06-12 01:05:24 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:24 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.09 seconds
|
||
[36m(AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304987) WARNING 06-12 01:05:24 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:19 [default_loader.py:293] Loading weights took 15.15 seconds[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:24 [vllm.py:690] Asynchronous scheduling is enabled.
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) WARNING 06-12 01:05:24 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304985) INFO 06-12 01:05:24 [vllm.py:846] Cudagraph is disabled under eager mode
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m WARNING 06-12 01:05:25 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m :441948 [0] NCCL INFO Broadcast: opCount 187 sendbuff 0x400e9fa02a00 recvbuff 0x400e9fa02a00 count 128 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02a40 count 32 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02a80 count 32 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa02ac0 count 32 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 188 sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401641800000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401643000000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401644800000 count 12582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 189 sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401641800000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401643000000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401644800000 count 12582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18a sendbuff 0x401640000000 recvbuff 0x401640000000 count 50331648 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401641800000 count 12582912 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401643000000 count 12582912 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401644800000 count 12582912 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18b sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5ac00 count 1024 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5b400 count 1024 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5bc00 count 1024 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18c sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5ac00 count 1024 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5b400 count 1024 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5bc00 count 1024 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18d sendbuff 0x400e9fa5a400 recvbuff 0x400e9fa5a400 count 4096 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5ac00 count 1024 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5b400 count 1024 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x400e9fa5bc00 count 1024 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x401660000000 recvbuff 0x401660000000 count 622329856 datatype 9 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Broadcast: opCount 18e sendbuff 0x401660000000 recvbuff 0x401660000000 count 622329856 datatype 9 op 0 root 0 comm 0x400dfa984d90 [nranks=2] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4016728c0000 count 155582464 datatype 9 op 0 root 1 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401685180000 count 155582464 datatype 9 op 0 root 2 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x401697a40000 count 155582464 datatype 9 op 0 root 3 comm 0x400dfaa9fd00 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:301668 [0] NCCL INFO cudaDriverVersion 13020
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:301668 [0] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.163<0>
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Initialized NET plugin IB
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Assigned NET plugin IB to comm
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Using network IB
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO DMA-BUF is available on GPU device 0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab0c759090 rank 25 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init START
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Setting affinity for GPU 0 to 0-71
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO comm 0xaaab0c759090 rank 25 nRanks 49 nNodes 13 localRanks 4 localRank 0 MNNVL 0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Trees [0] 38/-1/-1->25->21 [1] -1/-1/-1->25->40 [2] 38/-1/-1->25->39 [3] 39/-1/-1->25->40 [4] 38/21/26->25->17 [5] -1/-1/-1->25->40 [6] 38/-1/-1->25->39 [7] 39/-1/-1->25->40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO P2P Chunksize set to 131072
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO [Proxy Service] Device 0 CPU core 24
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302095 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 25
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302099 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 26
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 00/0 : 25[0] -> 26[0] [send] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 25[0] -> 26[0] [send] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 01/0 : 21[0] -> 25[0] [receive] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 05/0 : 21[0] -> 25[0] [receive] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 01/0 : 25[0] -> 38[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790800000 size 2097152 ipcDesc 0x40096400ae40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 05/0 : 25[0] -> 38[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790a00000 size 2097152 ipcDesc 0x40096400bc60
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 02/0 : 25[0] -> 40[3] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790c00000 size 2097152 ipcDesc 0x40096400ca30
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 03/0 : 25[0] -> 40[3] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401790e00000 size 2097152 ipcDesc 0x40096400d800
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 06/0 : 25[0] -> 40[3] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791000000 size 2097152 ipcDesc 0x40096400e5d0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 07/0 : 25[0] -> 40[3] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791200000 size 2097152 ipcDesc 0x40096400f3a0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791400000 size 10485760 ipcDesc 0x400964010170
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401791e00000 size 10485760 ipcDesc 0x400964010f40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401792800000 size 10485760 ipcDesc 0x400964010f40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401793200000 size 10485760 ipcDesc 0x400964010f40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401793c00000 size 10485760 ipcDesc 0x400964010f40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401794600000 size 10485760 ipcDesc 0x400964010f40
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401795000000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401795a00000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401796400000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401796e00000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401797800000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x401798200000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401798c00000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401798e00000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799000000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799200000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799400000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Imported shareable buffer device 0 size 2097152 ptr 0x401799600000
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 00/0 : 21[0] -> 25[0] [receive] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 21[0] -> 25[0] [receive] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 17[0] -> 25[0] [receive] via NET/IB/0
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 00/0 : 25[0] -> 38[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799800000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 02/0 : 25[0] -> 38[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799a00000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 04/0 : 25[0] -> 38[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799c00000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 06/0 : 25[0] -> 38[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x401799e00000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 02/0 : 25[0] -> 39[2] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a000000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 03/0 : 25[0] -> 39[2] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a200000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 06/0 : 25[0] -> 39[2] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a400000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 07/0 : 25[0] -> 39[2] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a600000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 01/0 : 25[0] -> 40[3] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179a800000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302065 [0] NCCL INFO Channel 05/0 : 25[0] -> 40[3] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179aa00000 size 2097152 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179ac00000 size 10485760 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179b600000 size 10485760 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-029-35:301668:302091 [0] NCCL INFO Allocated shareable buffer 0x40179c000000 size 10485760 ipcDesc 0x400964016140
|
||
[36m(AsyncVLLMInferenceEngine pid=301117, ip=10.128.25.163)[0m jpbo-0
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:20 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.667732 seconds[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:23 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:23 [kv_cache_utils.py:1307] GPU KV cache size: 359,120 tokens[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:23 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163)[0m jpb
|
||
[36m(AsyncVLLMInferenceEngine pid=315936, ip=10.128.25.23)[0m jpbo-026-39:316481:316921 [0] NCCL INFO Allocated shareable buffer 0x4017bd000000
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m jpbo-029-33:349999:350525 [0] NCCL INFO Allocated shareable buffer 0x4017bc600000 size 1
|
||
[36m(AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23)[0m jpbo-026-39:316471:316887 [0] NCCL INFO Channel 00/0 : 4[2] ->
|
||
[36m(AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23)[0m jpbo-026-39:316483:316889 [0] NCCL INFO Channel 00/0
|
||
[36m(AsyncVLLMInferenceEngine pid=349500, ip=10.128.25.161)[0m jpbo-029-33:349997:35
|
||
[36m(AsyncVLLMInferenceEngine pid=349498, ip=10.128.25.161)[0m jp
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m jpbo-026-39:316475:3
|
||
[36m(AsyncVLLMInferenceEngine pid=300569, ip=10.128.25.162)[0m jpbo-
|
||
[36m(AsyncVLLMInferenceEngine pid=300697, ip=10.128.25.162)[0m jpbo-0
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m jpbo-026-44:317118:317547 [0] NCCL INFO Allocat
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m jpbo-02
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m jpbo-0
|
||
[36m(AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165)[0m jpbo-0
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m jpbo-029-48:479197:4
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m jpbo-029-48:479189:479707 [0] NCCL INFO Allocated shareable buffer 0x4017bfc00000 size 2097152 ipcDes
|
||
[36m(AsyncVLLMInferenceEngine pid=478791, ip=10.128.25.176)[0m jpbo-029-48:47919
|
||
[36m(AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176)[0m jpbo-029-48:479185:479704 [0] NCCL INFO Allocated shareable buffer 0x40179e200000 s
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:327159 [0] NCCL INFO Imported shareable buffer de
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:250073 [0] NCCL INFO Allocated
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m jpbo-02
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-026-41:380145:380620 [0] NCCL INFO Connected all trees
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-026-41:380145:380620 [0] NCCL INFO threadThresholds 8/8/64 | 392/8/64 | 512 | 512
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-026-41:380145:380620 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 1 p2p channels per peer
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-026-41:380145:380620 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-026-41:380145:380620 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab14413a40 rank 5 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init COMPLETE
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-026-41:380145:380620 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 5 nranks 49 total 3.38 (kernels 0.32, alloc 2.10, bootstrap 0.28, allgathers 0.01, topo 0.03, graphs 0.01, connections 0.63, rest 0.01)
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m 33f250
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m 095833f6f0
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m 0485760 ipcDesc 0x40097033fc50
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=315936, ip=10.128.25.23)[0m size 10485760 ipcDesc 0x40097833bd20
|
||
[36m(AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23)[0m 3[1] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23)[0m : 3[1] -> 2[0] via P2P/IPC
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m d20
|
||
[36m(AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162)[0m 433fbc0
|
||
[36m(AsyncVLLMInferenceEngine pid=316510, ip=10.128.25.28)[0m ed shareable buffer 0x40179c600000 size 10485760 ipcDesc 0x40095c33fd50
|
||
[36m(AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28)[0m 5033d590
|
||
[36m(AsyncVLLMInferenceEngine pid=316639, ip=10.128.25.28)[0m 96833fda0
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m 33fd70
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m 833fc30
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m c 0x4009781d4530
|
||
[36m(AsyncVLLMInferenceEngine pid=478790, ip=10.128.25.176)[0m ize 2097152 ipcDesc 0x400970015eb0
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m c60
|
||
[36m(AsyncVLLMInferenceEngine pid=326287, ip=10.128.25.27)[0m 433eca0
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m vice 0 size 2097152 ptr 0x4017be200000
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m shareable buffer 0x40179c600000 size 10485760 ipcDesc 0x40096833fcd0
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m 3fc90
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 95433fe50
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount ff sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401820000000 recvbuff 0x401660000000 count 155582464 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 00/08 : 0 16 15 14 1 2 3 4 13 8 7 6 5 9 10 11 12 20 19 18 17 21 22 23 24 40 39 38 25 26 27 28 37 32 31 30 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 01/08 : 0 1 14 15 16 13 4 3 2 5 6 7 8 12 11 10 9 17 18 19 20 24 23 22 21 25 38 39 40 37 28 27 26 29 30 31 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 02/08 : 0 14 1 16 15 4 2 13 3 6 5 8 7 11 9 12 10 18 17 20 19 23 21 24 22 38 25 40 39 28 26 37 27 30 29 32 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 03/08 : 0 15 1 16 14 3 2 13 4 7 5 8 6 10 9 12 11 19 17 20 18 22 21 24 23 39 25 40 38 27 26 37 28 31 29 32 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 04/08 : 0 16 15 14 1 2 3 4 13 8 7 6 5 9 10 11 12 20 19 18 17 21 22 23 24 40 39 38 25 26 27 28 37 32 31 30 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 05/08 : 0 1 14 15 16 13 4 3 2 5 6 7 8 12 11 10 9 17 18 19 20 24 23 22 21 25 38 39 40 37 28 27 26 29 30 31 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 06/08 : 0 14 1 16 15 4 2 13 3 6 5 8 7 11 9 12 10 18 17 20 19 23 21 24 22 38 25 40 39 28 26 37 27 30 29 32 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 07/08 : 0 15 1 16 14 3 2 13 4 7 5 8 6 10 9 12 11 19 17 20 18 22 21 24 23 39 25 40 38 27 26 37 28 31 29 32 ...
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO CC Off, workFifoBytes 1048576
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 0 sendbuff 0x401660000000 recvbuff 0x401660000000 count 622329856 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 100 sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401635c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 1 sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 101 sendbuff 0x400e9
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [core.py:278] init engine (profile, create kv cache, warmup model) took 4.08 seconds[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opC
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m INFO Broadcast: opCount fa sendbuff 0x400e99e2b600 recvbuff 0x400e99e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2b800 count 32 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2b600 count 32 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010f8a00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010fa200000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x4010fba00000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2ba00 count 1024 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2c200 count 1024 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x400e99e2f600 count 1024 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x40116a400000 count 155582464 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4012f3e00000 recvbuff 0x401760000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] strea
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m tream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: op
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m fa88c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount a sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount b sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount c sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount f sendbuff 0x401660000000 recvbuff 0x401660000000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO Al
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m nranks=4] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m ount 108 sendbuff 0x400edfa88c00 recvbuff 0x400edfa88c00 count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendb
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m lGather: opCount 0 sendbuff 0x400e9fa5b600 recvbuff 0x400e9fa88e00 count 32 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m m 0x400e113f9600
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m 0x400e093f6c80
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m Count 0 sendbuff 0x4012d3a00000 recvbuff 0x4017aa400000 count 1048576 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ount 0 sendbuff 0x4012d3a00000 recvbuff 0x4017aa400000 count 1048576 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m jpbo-029-46:443488:443488 [0] NCCL INFO Broadcast: opCount 2a sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab28ec7140 [nran
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m jpbo-029-46:443505:443505 [0] NCCL INFO Broadcast: opCount 1c sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab032e5280 [nranks=49] s
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) WARNING 06-12 01:05:24 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401860000000 recvbuff 0x4016a0000000 count 12582912 datatype 9 o
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m uff 0x400eb9e2ea00 recvbuff 0x400eb9e88e00 count 1024 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 2c sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400e99e32c00 recvbuff 0x400e99
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 13a sendbuff 0x400e9fa8
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) WARNING 06-12 01:05:24 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m (EngineCore_DP0 pid=304986) INFO 06-12 01:05:24 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4010f4a0
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e412e5990 [
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [n
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m e88c00 count 32 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4] stream 0x400e113f9600
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCoun
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m WARNING 06-12 01:05:25 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 3a sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)[32m [repeated 1717x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x4016d7a40000 count 155582464 datatype 9 op 0 root 3 comm 0x400e06a9e080 [nranks=4] stream (nil)[32m [repeated 15x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)[32m [repeated 4x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x4010e6000000 recvbuff 0x4010e6000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)[32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x4010e0000000 recvbuff 0x4010e0000000 count 50331648 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x401100000000 recvbuff 0x401100000000 count 622329856 datatype 9 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)[32m [repeated 20x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)[32m [repeated 240x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m p 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 148 sendbuf
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 8c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m 0000 recvbuff 0x401718c00000 count 1048576 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:44
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4012e9400000 recvbuff 0x4017aa400000 count 4194304 datatype
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m jpbo-029-37:304985:304985 [0] NCCL INFO Broadcast: opCount 55 sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root
|
||
[36m(AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165)[0m jpbo-029-37:304987:304987 [0] NCCL INFO Broadcast: opCount 37 sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m : opCount 0 sendbuff 0x40184e400000 recvbuff 0x401660000000 count 12582912 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m t fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:44
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m f 0x400edfa89000 recvbuff 0x400edfa89000 count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 r
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 65 sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 dat
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:3649
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m atype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 173 sendbuff 0x400e9
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400faa000000 recvbuff 0x401760000000 count 4194304 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks=4
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: o
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400edfa67a00 recvbuff 0x400edfa88e00 count 32 datatype 9 op 0 root 0
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ecvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e3ae00 recvbu
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m jpbo-029-46:443488:443488 [0] NCCL INFO Broadcast: opCount 80 sendbuff 0x401720000000 recvbuff 0x401720000000 count 503
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m jpbo-029-46:443505:443505 [0] NCCL INFO Broadcast: opCount 52 sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 da
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m ] stream 0x400e113f9600
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 o
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m stream 0x400e093f6c80
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m pCount 0 sendbuff 0x400e99e3ae00 recvbuff 0x400e99e8b000 count 1024 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-02
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m INFO AllGather: opCount 0 sendbuff 0x40186e800000 recvbuff 0x401660000000 count 12582912 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401870800000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] str
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m ff 0x400eb9e8b000 count 1024 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e3e000 recvbuf
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e412e59
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:36
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m eam 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m p 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m 31648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x4003bdd82400 r
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m tatype 9 op 0 root 0 comm 0xaaab2a5a1f40 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m jpbo-026-40:249633:249633 [0] NCCL INFO Broadcast: opCount 6e sendbuff 0x4003bdd82400 recvbuff
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m f 0x400eb9e88c00 count 32 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 128 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCo
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401325c00000 recvbuff 0x401760000000 count 12582912 data
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401328c00000 recvbuff 0x401760000000 count 12582912 datat
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401666000000 recvbuff 0x401666000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount af sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m ount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:4423
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m unt fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:44231
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount be sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x401660000000 recvbuff 0x401660000000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x40188e200000 recvbuff 0x401913c00000 count 1048576 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [n
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 1c8 sendbuff 0x400edfa88e00 r
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL IN
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m type 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x4016d9800000 recvbuff 0x4016d9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab02e90db0 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m jpbo-026-40:249632:249632 [0] NCCL INFO Broadcast: opCo
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab2a5a1f40 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m jpbo-026-40:249633:249633 [0] NCCL INFO Broadcast: opCount 8a s
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m FO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401336400000 recvbuff 0x4017aa400000 count 4194304 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=4] s
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opC
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount da sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount db sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x400e9fa8b000 recvbuff 0x400e9fa8b000 count 4096 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m e5 sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401913c00000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x401660000000 recvbuff 0x401660000000 count 16777216 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x400e9fa88e00 recvbuff 0x400e9fa88e00 count 128 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 128 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401660000000 recvbuff 0x401660000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x401666000000 recvbuff 0x401666000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:44
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m tream 0x400e093f6c80
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m ount 0 sendbuff 0x40133e400000 recvbuff 0x401766000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m ] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401339c00000 recvbuff 0x401760000000 count 1258
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m unt d6 sendbuff 0x4003ddd82400 recvbuff 0x4003ddd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab1bf56550 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m jpbo-029-33:349999:349999 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab1bf56550 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=349371, ip=10.128.25.161)[0m jpbo-029-33:349999:349999 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab1bf56550 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161)[0m endbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab1d4cfe60 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161)[0m jpbo-029-33:350003:350003 [0] NCCL I
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nrank
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 208 sendbuff 0x400edfa88e00 recvbuff 0x400e
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [nr
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 110 sendbuff 0x401666000000 recvb
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m 2912 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: op
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m uff 0x401666000000 count 50331648 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0]
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllGat
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m Count 18f sendbuff 0x400eb9e89000 recvbuff 0x400eb9e89000 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce
|
||
[36m(AsyncVLLMInferenceEngine pid=304439, ip=10.128.25.165)[0m jpbo-029-37:304985:304985 [0] NCCL INFO Broadcast: opCount 12b sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 c
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m NFO Broadcast: opCount a5 sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab063e0f80 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=478791, ip=10.128.25.176)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=249227, ip=10.128.25.24)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=379729, ip=10.128.25.25)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=379730, ip=10.128.25.25)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=305554, ip=10.128.25.164)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=443231, ip=10.128.25.174)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=301250, ip=10.128.25.163)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=349499, ip=10.128.25.161)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=349500, ip=10.128.25.161)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=316064, ip=10.128.25.23)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=316063, ip=10.128.25.23)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=300698, ip=10.128.25.162)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=300697, ip=10.128.25.162)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=316638, ip=10.128.25.28)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=304568, ip=10.128.25.165)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029
|
||
[36m(AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=326287, ip=10.128.25.27)[0m jpbo-026
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m Count fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:4
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m her: opCount 0 sendbuff 0x400edfa7b200 recvbuff 0x400edfa8b000 count 1024 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400edfa7d200 recvbuff 0x400edfa88e00 count 1024 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] st
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m : opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (n
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m ream 0x400e067c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 248 sendbuff 0x400edfa88e00 recvbuff 0x400edfa88e00
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:44260
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401391400000 recvbuff 0x401760000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e116cc830 [nranks
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401374400000 recvbuff 0x401760000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllGath
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m omm 0xaaab25495c20 [nranks=49] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=478659, ip=10.128.25.176)[0m jpbo-029-48:479197:479197 [0] NCCL INFO Broadcast: opCount 156 sendbuff 0x4016d
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m jpbo-029-48:479189:479189 [0] NCCL INFO Broadcast: opCount da sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab063e0f80 [nran
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m il)
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e53e00 recvbuff 0x400eb9e8b000 count 1024 dataty
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m t 18f sendbuff 0x400eb9e88e00 recvbuff 0x400eb9e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m j
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m er: opCount 0 sendbuff 0x400e99e53200 recvbuff 0x400e99e88e00 count 32 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m pe 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 265 sendbuff 0x400e9fa88c00 recv
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m buff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x400dfa6b40c0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NC
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m 0 sendbuff 0x401900000000 recvbuff 0x4016a0000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401379800000 recvbuff 0x4017aa400000 count 1048576 datatype 9 op
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:443494 [0] NCCL INFO cudaDriverVersion 13020[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:443494 [0] NCCL INFO NCCL version 2.27.7+cuda13.0[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. [32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.25.174<0>[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO Initialized NET plugin IB[32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Assigned NET plugin IB to comm[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Using network IB[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO DMA-BUF is available on GPU device 0[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO ncclCommInitRankConfig comm 0x400ef44ecc80 rank 0 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init START[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>[32m [repeated 47x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC[32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Setting affinity for GPU 0 to 0-71[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)[32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO comm 0x400ef44ecc80 rank 0 nRanks 49 nNodes 13 localRanks 1 localRank 0 MNNVL 0[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Trees [0] 26/-1/-1->0->-1 [1] 27/-1/-1->0->-1 [2] 28/-1/-1->0->-1 [3] 37/-1/-1->0->-1 [4] 41/-1/-1->0->29 [5] 42/-1/-1->0->30 [6] 43/-1/-1->0->31 [7] 44/-1/-1->0->32[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO P2P Chunksize set to 131072[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m jpbo-029-46:443494:444068 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. [32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442612 [0] NCCL INFO [Proxy Service] Device 0 CPU core 51[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442613 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 52[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442614 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 54[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 07/0 : 0[0] -> 44[3] [send] via NET/IB/0[32m [repeated 282x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Channel 00/0 : 26[0] -> 0[0] [receive] via NET/IB/0[32m [repeated 279x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:250043 [0] NCCL INFO Channel 06/0 : 22[1] -> 21[0] via P2P/IPC[32m [repeated 654x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:250071 [0] NCCL INFO Allocated shareable buffer 0x40177e600000 size 2097152 ipcDesc 0x40094433fc90[32m [repeated 1312x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m jpbo-026-40:249637:250044 [0] NCCL INFO Imported shareable buffer device 0 size 10485760 ptr 0x4016dd800000[32m [repeated 1331x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27)[0m jpbo-026-43:326764:327196 [0] NCCL INFO Allocated shareable buffer 0x40177e000000 size 2097152
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jp[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379601, ip=10.128.25.25)[0m jpbo-
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Connected all trees[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO threadThresholds 8/8/64 | 392/8/64 | 512 | 512[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 1 p2p channels per peer[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m jpbo-026-40:249637:250044 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.[32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO ncclCommInitRankConfig comm 0x400ef44ecc80 rank 0 nranks 49 cudaDev 0 nvmlDev 0 busId 901000 commId 0xa8aa92223a1db3f3 - Init COMPLETE[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442611 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 49 total 3.47 (kernels 0.00, alloc 0.00, bootstrap 2.79, allgathers 0.04, topo 0.01, graphs 0.00, connections 0.63, rest 0.00)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326288, ip=10.128.25.27)[0m ipcDesc 0x40094433fc30
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m CL INFO AllGather: opCount 0 sendbuff 0x4018ee800000 recvbuff 0x401660000000 count 12582912 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] stream 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCo
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m 67c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount 288 sendbuff 0x400edfa88e00 recvbuff 0x400edfa88e00 count 1 datatyp
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520
|
||
[36m(AsyncVLLMInferenceEngine pid=300569, ip=10.128.25.162)[0m (EngineCore_DP0 pid=301120) INFO 06-12 01:05:34 [block_pool.py:452] Successfully reset prefix cache
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllReduce: opCount ff sendbuff 0x400edfa88c00 recvbuff 0x400edfa88c00 count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x40138a400000 recvbuff 0x401760000000 count 155582464 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520[32m [repeated 3044x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e252e2e80 [nranks=8] stream (nil)[32m [repeated 1532x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCount 18f sendbuff [32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatyp
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: opC[32m [repeated 3x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:05:44] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:05:44] OK: RSS 0.89 GiB | node mem 381.3/858.0 GiB used (44.4%), avail 476.6 GiB
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m INFO Broadcast: opCount fa sendbuff 0x400e99e2b600 recvbuff 0x400e99e2b600 count 128 datatype 9 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x40114a400000 count 155582464 datatype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream (nil)[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400e9fa64200 recvbuff 0x400e9fa88e00 count 1024 datatype 9 op 0 root 0 comm 0x400dfaa9fd00 [nranks=4] strea
|
||
[36m(AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28)[0m tream (nil)[32m [repeated 25x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllGather: op[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m 9800000 recvbuff 0x4016d9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab0d165c60 [nranks=49] stream (nil)[32m [repeated 31x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount a sendbuff 0x40079203c400 recvbuff 0x40079203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount b sendbuff 0x40079203c400 recvbuff 0x40079203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount c sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount d sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount e sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount f sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m 4700 [nranks=8] stream (nil)[32m [repeated 88x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m op 0 root 0 comm 0x400e112e5db0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m m 0x400dfa7c8310
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m Count 0 sendbuff 0x40133b400000 recvbuff 0x401766000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160[32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 18f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m jpbo-029-36:305972:305972 [0] NCCL INFO Broadcast: opCount 2a sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab1d7661e0 [nran[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164)[0m jpbo-029-36:305976:305976 [0] NCCL INFO Broadcast: opCount 1c sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab0c5666e0 [nranks=49] s[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443364, ip=10.128.25.174)[0m 0 comm 0xaaab132435a0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x40138a000000 recvbuff 0x4017[32m [repeated 10x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)[32m [repeated 5x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4012e74000
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e89000 recvbuff 0x400e99e89000 count 1 datatype 7 op 0 root 0 comm 0x400e412e5990 [n
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m 66000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e096cb190 [nranks=4] stream 0x400e093f6c80[32m [repeated 10x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllReduce: opCoun
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m jpbo-026-40:249637:249637 [0] NCCL INFO Broadcast: opCount 180 sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab0d165c60 [nranks=49] stream (nil)[32m [repeated 14484x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fa sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fb sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fc sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fd sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0 root 0 comm 0x400e412e4700 [nranks=8] stream (nil)[32m [repeated 1323x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllReduce: opCount 1[32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 00 recvbuff 0x401766000000 count 12582912 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400e99e4e400 recvbuff 0x400e99e88c00 count 32 datatype 9[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount f6 sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root [32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount 157 sendbuff 0x401389580000 recvbuff 0x401389580000 count 4194304 datatype 9 op 0 root 0 comm 0[32m [repeated 25x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllReduce: opCount 22c sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 1 datatype 7 op 0 root 0 comm 0x40[32m [repeated 5x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:441948 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4018f0800000 recvbuff 0x401913c00000 count 4194304 datatype 9 op 0 root 0 comm 0x400dfaa9fd0[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m ecvbuff 0x40039dd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab0aad8260 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m jpbo-029-44:441858:442603 [0] NCCL INFO Broadcast: opCount d7 sendbuff 0x400e9fa88c00 recvbuff 0x400e9fa88c00 count 12[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164)[0m jpbo-029-36:305976:305976 [0] NCCL INFO Broadcast: opCount 52 sendbuff 0x401740000000 recvbuff 0x401740000000 count 50331648 da[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL [32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 o
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m 4] stream 0x400e093f6c80[32m [repeated 7x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m jpbo-029-44:441955:442310 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88e00 recvbuff 0x400e99e88e00 count 1 datatype 7 op 0[32m [repeated 2x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m INFO AllGather: opCount 0 sendbuff 0x4018b1000000 recvbuff 0x401389180000 count 1048576 datatype 9 op 0 root 0 comm 0x400e06a9e080 [nranks=4] stream 0x400e067c71c0
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e412e[32m [repeated 3x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m p 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)
|
||
[36m(FSDPPolicyWorkerBase pid=441858, ip=10.128.25.172)[0m 8 datatype 9 op 0 root 0 comm 0x400ef44ecc80 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount aa sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m jpbo-026-43:326770:326770 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x40039dd82400 r[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m tatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount 6e sendbuff 0x4003ddd82400 recvbuff [32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ab sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 25x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ac sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m jpbo-029-44:441954:442312 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x400eb9e50a00 recvbuff 0x400eb9e88e00 count 32 datat
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ad sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ae sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount af sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount ba sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [[32m [repeated 23x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=441955, ip=10.128.25.172)[0m ount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e092e33d0 [nranks=8] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bb sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bc sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bd sendbuff 0x401760000000 recvbuff 0x401760000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount be sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount bf sendbuff 0x401759200000 recvbuff 0x401759200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m jpbo-026-33:364946:365413 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4018c2000000 recvbuff 0x401932a00000 count 4194304 datatype 9 op 0 root 0 comm 0x400e06a9e080 [n
|
||
[36m(FSDPPolicyWorkerBase pid=441954, ip=10.128.25.172)[0m ype 9 op 0 root 0 comm 0x400e256ca1f0 [nranks=4] stream 0x400e253f6520
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount ca sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cb sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cc sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cd sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount ce sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount cf sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326155, ip=10.128.25.27)[0m jpbo-026-43:326770:326770 [0] NCCL INFO Broadcast: opCo[32m [repeated 22x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364946)[0m count 1 datatype 7 op 0 root 0 comm 0x400e066b2ed0 [nranks=8] stream (nil)[32m [repeated 26x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m jpbo-026-43:326772:326772 [0] NCCL INFO Broadcast: opCount 8a s[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount da sendbuff 0x40075203c400 recvbuff 0x40075203c400 count 4096 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nranks=49] stream (nil)[32m [repeated 73x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount db sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount dc sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount dd sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount de sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount df sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 8f sendbuff 0x400eb9e88c00 recvbuff 0x400eb9e88c00 count 1 datatype 7 op 0 root 0 comm 0x400e612dc480 [nranks=8] stream (nil)
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ea sendbuff 0x401719200000 recvbuff 0x401719200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount eb sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ec sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ed sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ee sendbuff 0x401720000000 recvbuff 0x401720000000 count 50331648 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 48x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249225, ip=10.128.25.24)[0m jpbo-026-40:249625:249625 [0] NCCL INFO Broadcast: opCount ef sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab168db1e0 [nranks=49] stream (nil)[32m [repeated 98x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo-026-33:364945:365415 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x401342a00000 recvbuff 0x40171a400000 count
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m unt d6 sendbuff 0x4003bdd82400 recvbuff 0x4003bdd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount fe sendbuff 0x4016d9800000 recvbuff 0x4016d9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m jpbo-029-35:301664:301664 [0] NCCL INFO Broadcast: opCount ff sendbuff 0x401739200000 recvbuff 0x401739200000 count 4194304 datatype 9 op 0 root 0 comm 0xaaab2ad55ce0 [nranks=49] stream (nil)[32m [repeated 23x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m endbuff 0x4003ddd82400 recvbuff 0x4003ddd82400 count 128 datatype 9 op 0 root 0 comm 0xaaab3afe1e00 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m jpbo-029-35:301662:301662 [0] NCCL I[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m 1048576 datatype 9 op 0 root 0 comm 0x400e616c3ee0 [nranks=4] stream 0x400e613efdc0
|
||
[36m(FSDPPolicyWorkerBase pid=441953, ip=10.128.25.172)[0m jpbo-029-44:441953:442311 [0] NCCL INFO AllReduce: op
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m jpbo-026-33:364947:365414 [0] NCCL INFO AllGat
|
||
[36m(AsyncVLLMInferenceEngine pid=316639, ip=10.128.25.28)[0m jpbo-026-44:317124:317124 [0] NCCL INFO Broadcast: opCount 12b sendbuff 0x401739200000 recvbuff 0x401739200000 count 4194304 datatype 9 op 0 root 0 c[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m NFO Broadcast: opCount a5 sendbuff 0x4016f9800000 recvbuff 0x4016f9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaaaf0c213c0 [nranks=49] stream (nil)[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364947)[0m her: opCount 0 sendbuff 0x400e99e53200 recvbuff 0x400e99e88e00 count 32 datatype 9 op 0 root 0 comm 0x400e416caff0 [nranks=4] stream 0x400e413f9160
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m omm 0xaaab5583a610 [nranks=49] stream (nil)[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304567, ip=10.128.25.165)[0m jpbo-029-37:304986:304986 [0] NCCL INFO Broadcast: opCount 156 sendbuff 0x4016f[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=304566, ip=10.128.25.165)[0m jpbo-029-37:304984:304984 [0] NCCL INFO Broadcast: opCount da sendbuff 0x4016b9800000 recvbuff 0x4016b9800000 count 16777216 datatype 9 op 0 root 0 comm 0xaaab3136e390 [nran[32m [repeated 24x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllReduce: opCount fe sendbuff 0x400e99e88c00 recvbuff 0x400e99e88c00 count 1 datatype
|
||
[36m(FSDPPolicyWorkerBase pid=364945)[0m jpbo
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m jpbo-[32m [repeated 22x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m (EngineCore_DP0 pid=316475) INFO 06-12 01:05:34 [block_pool.py:452] Successfully reset prefix cache[32m [repeated 47x across cluster][0m
|
||
[36m(FSDPPolicyWorkerBase pid=364944)[0m jpbo-026-33:364944:365416 [0] NCCL INFO AllGather: opCount 0 sendbuff 0x4013a0000000 recvbuff 0x401760000000 count 155582464 datatype 9 op 0 root 0 comm 0x400e416cc220 [nranks=4] stream 0x400e413f8020[32m [repeated 42x across cluster][0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:05:51] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:05:51] OK: RSS 0.89 GiB | node mem 368.5/858.0 GiB used (42.9%), avail 489.5 GiB
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:05:58] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:05:58] OK: RSS 0.89 GiB | node mem 379.8/858.0 GiB used (44.3%), avail 478.2 GiB
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:06:07] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)[32m [repeated 2x across cluster][0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:06:07] OK: RSS 0.89 GiB | node mem 389.9/858.0 GiB used (45.4%), avail 468.0 GiB[32m [repeated 2x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:07:44] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:07:44] OK: RSS 0.89 GiB | node mem 381.5/858.0 GiB used (44.5%), avail 476.5 GiB
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:07:51] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:07:51] OK: RSS 0.89 GiB | node mem 368.5/858.0 GiB used (43.0%), avail 489.4 GiB
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:07:58] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:07:58] OK: RSS 0.89 GiB | node mem 379.9/858.0 GiB used (44.3%), avail 478.0 GiB
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [01:08:01] OK: 104 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [01:08:01] OK: RSS 1.74 GiB | node mem 245.2/858.0 GiB used (28.6%), avail 612.8 GiB
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:08:07] OK: 47 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:08:07] OK: RSS 0.89 GiB | node mem 390.0/858.0 GiB used (45.5%), avail 468.0 GiB
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 3x across cluster][0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 714x across cluster][0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 797x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=326289, ip=10.128.25.27)[0m INFO 06-12 01:09:13 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:18 [loggers.py:259] Engine 000: Avg prompt throughput: 2.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
||
[36m(AsyncVLLMInferenceEngine pid=305555, ip=10.128.25.164)[0m INFO 06-12 01:09:17 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 9x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%[32m [repeated 14x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301251, ip=10.128.25.163)[0m INFO 06-12 01:09:23 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 9x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 21x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249097, ip=10.128.25.24)[0m INFO 06-12 01:09:25 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 4x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%[32m [repeated 29x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=300699, ip=10.128.25.162)[0m INFO 06-12 01:09:31 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 6x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:38 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%[32m [repeated 30x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=305556, ip=10.128.25.164)[0m INFO 06-12 01:09:34 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:43 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%[32m [repeated 28x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:09:44] OK: 263 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [fd-monitor] [01:09:44] OK: RSS 1.13 GiB | node mem 382.1/858.0 GiB used (44.5%), avail 475.9 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%[32m [repeated 31x across cluster][0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:09:51] OK: 263 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [fd-monitor] [01:09:51] OK: RSS 1.13 GiB | node mem 369.2/858.0 GiB used (43.0%), avail 488.8 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%[32m [repeated 31x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:09:58 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%[32m [repeated 28x across cluster][0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:09:58] OK: 215 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [fd-monitor] [01:09:58] OK: RSS 1.07 GiB | node mem 380.3/858.0 GiB used (44.3%), avail 477.7 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:10:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%[32m [repeated 29x across cluster][0m
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [01:10:01] OK: 177 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
|
||
[36m(skyrl_entrypoint pid=364638)[0m [fd-monitor] [01:10:01] OK: RSS 1.76 GiB | node mem 245.3/858.0 GiB used (28.6%), avail 612.7 GiB
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:10:07] OK: 215 / 131,072 FDs open (0.2% of soft limit, hard limit: 131,072)
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m [fd-monitor] [01:10:07] OK: RSS 1.07 GiB | node mem 390.6/858.0 GiB used (45.5%), avail 467.4 GiB
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:10:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%[32m [repeated 28x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:10:13 [loggers.py:259] Engine 000: Avg prompt throughput: 190.6 tokens/s, Avg generation throughput: 26.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 64.5%[32m [repeated 27x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=301252, ip=10.128.25.163)[0m INFO 06-12 01:10:16 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m
|
||
[36m(RolloutCoordinator pid=479438, ip=10.128.25.176)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316189, ip=10.128.25.23)[0m INFO 06-12 01:10:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 64.5%[32m [repeated 30x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=379731, ip=10.128.25.25)[0m INFO 06-12 01:10:18 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 4x across cluster][0m
|
||
[36m(RolloutCoordinator pid=350242, ip=10.128.25.161)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 3x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:23 [loggers.py:259] Engine 000: Avg prompt throughput: 57.1 tokens/s, Avg generation throughput: 83.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 80.3%[32m [repeated 27x across cluster][0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(RolloutCoordinator pid=380389, ip=10.128.25.25)[0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443104, ip=10.128.25.174)[0m INFO 06-12 01:10:27 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m
|
||
[36m(RolloutCoordinator pid=301361, ip=10.128.25.162)[0m [1;31mProvider List: https://docs.litellm.ai/docs/providers[0m[32m [repeated 4x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:28 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 56.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 80.3%[32m [repeated 26x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=316640, ip=10.128.25.28)[0m INFO 06-12 01:10:31 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 5x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 80.3%[32m [repeated 35x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=478789, ip=10.128.25.176)[0m INFO 06-12 01:10:34 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.[32m [repeated 2x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:38 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 80.3%[32m [repeated 41x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:43 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 80.3%[32m [repeated 40x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 80.3%[32m [repeated 40x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=249226, ip=10.128.25.24)[0m INFO 06-12 01:10:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 80.3%[32m [repeated 38x across cluster][0m
|
||
[36m(AsyncVLLMInferenceEngine pid=443232, ip=10.128.25.174)[0m INFO 06-12 01:10:56 [loggers.py:259] Engine 000: Avg prompt throughput: 64.7 tokens/s, Avg generation throughput: 77.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 68.0%[32m [repeated 21x across cluster][0m
|
||
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=seq_mean_token_sum_norm_global', 'trainer.algorithm.use_tis=true', 'trainer.algorithm.tis_imp_ratio_cap=2.0', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/explore-tis-untrunc', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=explore-tis-untrunc', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.collect_rollout_details=true', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true', 'generator.sampling_params.temperature=1.0', 'generator.sampling_params.top_p=1.0', 'generator.sampling_params.top_k=-1']
|
||
Traceback (most recent call last):
|
||
File "<frozen runpy>", line 198, in _run_module_as_main
|
||
File "<frozen runpy>", line 88, in _run_code
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
|
||
main()
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
|
||
_run_hydra(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
|
||
_run_app(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
|
||
run_and_report(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
|
||
raise ex
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
|
||
return func()
|
||
^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
|
||
lambda: hydra.run(
|
||
^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
|
||
_ = ret.return_value
|
||
^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
|
||
raise self._return_value
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
|
||
ret.return_value = task_function(task_cfg)
|
||
^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
|
||
ray.get(skyrl_entrypoint.remote(cfg))
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
|
||
return fn(*args, **kwargs)
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
|
||
return func(*args, **kwargs)
|
||
^^^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
|
||
values, debugger_breakpoint = worker.get_objects(
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1017, in get_objects
|
||
raise value
|
||
ray.exceptions.NodeDiedError: Task failed because the node it was running on is dead or unavailable. Node IP: 10.128.25.17, node ID: 96706d0fc7d05b397467667c5e336ac21712cf7e0c8bf1b2f4037765. This can happen if the node was preempted, had a hardware failure, or its raylet crashed unexpectedly. To see node death information, use `ray list nodes --filter node_id=96706d0fc7d05b397467667c5e336ac21712cf7e0c8bf1b2f4037765`, check the Ray dashboard cluster page, search the node ID in the GCS logs, or use `ray logs raylet.out -ip 10.128.25.17`.
|
||
Stopping Ray cluster...
|
||
Ray cluster stopped
|
||
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/explore-tis-untrunc/explore-tis-untrunc/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|