icer/qwen36_01
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

test eager and custom allreduce toggles

Browse Source
This commit is contained in:
the magician
2026-07-14 19:18:46 +08:00
parent 82c1463b0c
commit a5abd66e21
7 changed files with 777 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
worklogs/decode_analysis_report_2026-07-14.md
View File

@@ -431,3 +431,107 @@ self._decode_wrapper = BatchDecodeWithPagedKVCacheWrapper(...)
- 直接在当前可用 xFormers 路径插桩。
- 重点记录每 token decode 中 attention、MoE、sampler、TP 同步的耗时。
- 当前平均 GPU 利用率低，profiling 比继续盲调参数更有价值。
## 2026-07-14：解除强制 eager / custom all-reduce 禁用验证
### 修改内容
本轮先处理 `vllm/engine/arg_utils.py` 中两个会掩盖真实性能路径的硬编码：
```python
enforce_eager=True
disable_custom_all_reduce=True
```
改为尊重 CLI / dataclass 参数：
```python
enforce_eager=self.enforce_eager
disable_custom_all_reduce=self.disable_custom_all_reduce
```
同时更新 `qwen3_6_scripts/patch_xformers_sdpa_seq.py`，让后续重新执行 patchops 时也会保留该行为。服务器实际运行路径已确认：
```text
PYTHONPATH=/usr/local/corex/lib/python3/dist-packages:/usr/local/corex/lib64/python3/dist-packages
```
因此运行时同时 patch 了：
- `/usr/local/corex/lib/python3/dist-packages/vllm/engine/arg_utils.py`
- `/usr/local/corex/lib64/python3/dist-packages/vllm/engine/arg_utils.py`
远端运行时备份：
- `arg_utils.py.bak_20260714_eager_allreduce`
### 实验 1：CUDA Graph + custom all-reduce 同时开启
启动命令不再带 `--enforce-eager`，也不带 `--disable-custom-all-reduce`。
日志确认配置已生效：
```text
disable_custom_all_reduce=False
enforce_eager=False
use_async_output_proc=True
```
结果：服务未能完成启动，长时间卡在 CUDA Graph capture 阶段。
关键日志：
```text
Capturing the model for CUDA graphs.
[W CUDAGraph.cpp:145] Warning: Waiting for pending NCCL work to finish before starting graph capture.
```
判断：当前 Iluvatar BI-V100 + CoreX + xFormers + TP=4 路径下，CUDA Graph capture 不可直接启用。它没有快速报错，而是卡在 graph capture / NCCL pending work 阶段，风险比普通参数不兼容更高。短期不建议继续沿 CUDA Graph 方向盲试。
本地归档：
- `worklogs/remote_results/2026-07-14-eager-allreduce/server_exp_graph_allreduce_seq2_b8192.log`
### 实验 2：仅开启 custom all-reduce，继续 eager
启动命令保留 `--enforce-eager`，但不再带 `--disable-custom-all-reduce`。
日志确认：
```text
enforce_eager=True
disable_custom_all_reduce=False
```
服务可以正常启动并通过 `/health`。
decode microbench 结果：
| 配置 | 成功率 | TTFT P90 | Output TPS P10/req | Aggregate Output TPS | 对比旧结果 |
| --- | ---: | ---: | ---: | ---: | --- |
| short c1, 256 tok, custom AR on | 100% | 3.47s | 8.47 | 7.94 | 旧 full-parser c1 为 P10 8.74 / aggregate 8.34，略降 |
| short c2, 128 tok, custom AR on | 100% | 1.46s | 4.22 | 8.07 | 旧 parser-off c2 aggregate 7.23，略升但口径不完全相同 |
本地归档：
- `worklogs/remote_results/2026-07-14-eager-allreduce/decode_eager_custom_ar_short_c1_t256_r3.json`
- `worklogs/remote_results/2026-07-14-eager-allreduce/decode_eager_custom_ar_short_c2_t128_r4.json`
- `worklogs/remote_results/2026-07-14-eager-allreduce/server_exp_eager_custom_ar_seq2_b8192_retry.log`
### 结论
1. 之前的硬编码确实屏蔽了真实配置，本轮已经解除，并确认修改落在实际运行的 CoreX site-packages 路径里。
2. CUDA Graph 当前不兼容或存在严重启动卡死问题，不适合作为短期主优化方向。
3. custom all-reduce 可以启动和推理，但收益有限：c2 聚合吞吐有小幅提升，c1 无提升。
4. 当前 decode 吞吐仍在 8 tok/s 左右，距离 Output TPS P10 >= 20 仍有明显差距，瓶颈不只是 all-reduce 开关。
### 下一步 profiling 方向
优先进入代码级 profiling，而不是继续调 CLI 开关：
1. 在 `ModelRunner.execute_model` 统计模型 forward、logits、sample 的阶段耗时。
2. 在 Qwen MoE 层统计 attention、MoE expert、MoE gate、TP all-reduce 的耗时占比。
3. 在 fused MoE 路径统计 topk、expert kernel、activation、sum 的耗时。
4. 用 `ENGINEX_PROFILE_DECODE=1` 这类环境变量控制插桩，只在短压测时开启，避免污染正式结果。
初步判断：custom all-reduce 不是第一大瓶颈；更可能的主战场是 xFormers decode attention、MoE 小 batch kernel、以及 TP 下大量小 kernel / 同步造成的低 GPU 利用率。

63
worklogs/remote_results/2026-07-14-eager-allreduce/decode_eager_custom_ar_short_c1_t256_r3.json Normal file
View File

@@ -0,0 +1,63 @@
{
"created_at": "2026-07-14T11:00:49",
"label": "eager_custom_ar_short_c1_t256_r3",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 3,
"max_tokens": 256,
"wall_sec": 96.7806523796171,
"success_rate": 1.0,
"ttft_p50_sec": 1.0768930949270725,
"ttft_p90_sec": 3.4729231126606463,
"output_tps_p10_per_request": 8.472860425455258,
"output_tps_p50_per_request": 8.483117808158335,
"aggregate_output_tps": 7.935470376739762,
"prompt_tokens": 117,
"cached_tokens": 64,
"completion_tokens": 768,
"reasoning_tokens": 768,
"chars": 2635,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 34.2951952572912,
"ttft_sec": 4.07193061709404,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 256,
"output_tps": 8.470296079779487,
"chars": 916,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.25290015526116,
"ttft_sec": 1.0768930949270725,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 256,
"output_tps": 8.483561111586171,
"chars": 856,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.230418637394905,
"ttft_sec": 1.05283466540277,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 256,
"output_tps": 8.483117808158335,
"chars": 863,
"error": null
}
]
}

75
worklogs/remote_results/2026-07-14-eager-allreduce/decode_eager_custom_ar_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,75 @@
{
"created_at": "2026-07-14T11:01:53",
"label": "eager_custom_ar_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 63.41567398421466,
"success_rate": 1.0,
"ttft_p50_sec": 1.449904115870595,
"ttft_p90_sec": 1.4555151607841255,
"output_tps_p10_per_request": 4.218399789762928,
"output_tps_p50_per_request": 4.2304733122443885,
"aggregate_output_tps": 8.073713765581775,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1816,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 31.78747241385281,
"ttft_sec": 1.4443142116069794,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 4.21841388911607,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.787043346092105,
"ttft_sec": 1.4437402617186308,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 4.2183937471830095,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.626181228086352,
"ttft_sec": 1.4555242210626602,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 4.242532735372708,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.625997802242637,
"ttft_sec": 1.4554940201342106,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 4.24255428163934,
"chars": 454,
"error": null
}
]
}

398
worklogs/remote_results/2026-07-14-eager-allreduce/server_exp_eager_custom_ar_seq2_b8192_retry.log Normal file
View File

@@ -0,0 +1,398 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 10:55:02 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 10:55:03.598198: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 10:55:03.649443: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 10:55:09 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 10:55:09 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=32768, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 10:55:09 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 10:55:20 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 10:55:20 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 07-14 10:55:20 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 07-14 10:55:20 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 10:55:20 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 10:55:20 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 10:55:20 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 10:55:20 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 10:55:22 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 10:55:22 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 10:55:22 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=9684) INFO 07-14 10:55:29 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=9684) INFO 07-14 10:55:29 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=9684) INFO 07-14 10:55:29 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=9683) INFO 07-14 10:55:29 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=9683) INFO 07-14 10:55:29 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=9683) INFO 07-14 10:55:29 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=9685) INFO 07-14 10:55:30 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=9685) INFO 07-14 10:55:30 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=9685) INFO 07-14 10:55:30 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 10:55:30 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f63d51f5a50>, local_subscribe_port=46879, remote_subscribe_port=None)
INFO 07-14 10:55:30 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=9684) INFO 07-14 10:55:30 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=9685) INFO 07-14 10:55:30 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=9683) INFO 07-14 10:55:30 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
INFO 07-14 10:55:30 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 10:55:30 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=9684) INFO 07-14 10:55:30 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=9684) INFO 07-14 10:55:30 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=9683) INFO 07-14 10:55:30 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=9683) INFO 07-14 10:55:30 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=9685) INFO 07-14 10:55:30 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=9685) INFO 07-14 10:55:30 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:44, 1.77s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:25, 1.05s/it]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:04<00:31, 1.39s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:06<00:35, 1.61s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:07<00:29, 1.38s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:08<00:31, 1.58s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:09<00:23, 1.21s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:11<00:24, 1.39s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:12<00:21, 1.25s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:14<00:23, 1.47s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:14<00:16, 1.12s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:16<00:18, 1.34s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:18<00:19, 1.50s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:20<00:20, 1.69s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:21<00:15, 1.43s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:23<00:17, 1.73s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:24<00:12, 1.38s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:26<00:12, 1.55s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:27<00:11, 1.60s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:30<00:11, 1.87s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:31<00:08, 1.62s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:32<00:05, 1.35s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:34<00:04, 1.65s/it]
(VllmWorkerProcess pid=9685) INFO 07-14 10:56:06 model_runner.py:1076] Loading model weights took 16.2303 GB
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:36<00:03, 1.76s/it]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:38<00:01, 1.82s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.43s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.50s/it]
(VllmWorkerProcess pid=9684) INFO 07-14 10:56:09 model_runner.py:1076] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=9683) INFO 07-14 10:56:09 model_runner.py:1076] Loading model weights took 16.2303 GB
INFO 07-14 10:56:09 model_runner.py:1076] Loading model weights took 16.2303 GB
INFO 07-14 10:56:17 distributed_gpu_executor.py:57] # GPU blocks: 21100, # CPU blocks: 6553
INFO 07-14 10:56:17 distributed_gpu_executor.py:61] Maximum concurrency for 100000 tokens per request: 3.38x
INFO 07-14 10:56:21 serving_chat.py:79] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
INFO 07-14 10:56:21 serving_chat.py:101] Reasoning parser 'qwen3' enabled.
WARNING 07-14 10:56:21 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
INFO 07-14 10:56:21 launcher.py:19] Available routes are:
INFO 07-14 10:56:21 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
INFO 07-14 10:56:21 launcher.py:27] Route: /docs, Methods: HEAD, GET
INFO 07-14 10:56:21 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 07-14 10:56:21 launcher.py:27] Route: /redoc, Methods: HEAD, GET
INFO 07-14 10:56:21 launcher.py:27] Route: /health, Methods: GET
INFO 07-14 10:56:21 launcher.py:27] Route: /tokenize, Methods: POST
INFO 07-14 10:56:21 launcher.py:27] Route: /detokenize, Methods: POST
INFO 07-14 10:56:21 launcher.py:27] Route: /v1/models, Methods: GET
INFO 07-14 10:56:21 launcher.py:27] Route: /version, Methods: GET
INFO 07-14 10:56:21 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 07-14 10:56:21 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 07-14 10:56:21 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO: Started server process [9342]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on socket ('0.0.0.0', 1111) (Press CTRL+C to quit)
INFO 07-14 10:56:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:56:31 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:44316 - "GET /health HTTP/1.1" 200 OK
INFO 07-14 10:56:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:56:41 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:56:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:56:51 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:57:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:57:01 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:57:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:57:11 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:57:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:57:21 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:57:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:57:31 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:57:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:57:41 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:49340 - "GET /health HTTP/1.1" 200 OK
INFO 07-14 10:57:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:57:51 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:58:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:58:01 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:58:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:58:11 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:58:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:58:21 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:58:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:58:31 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:58:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:58:41 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:58:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:58:51 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:01 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:11 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:35214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import resource_string
INFO 07-14 10:59:16 metrics.py:345] Avg prompt throughput: 7.7 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:16 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:21 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:26 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:31 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:36 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 10:59:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:42 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:41594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 10:59:48 metrics.py:345] Avg prompt throughput: 6.5 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:48 metrics.py:361] Prefix cache hit rate: GPU: 50.00%, CPU: 0.00%
INFO 07-14 10:59:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:53 metrics.py:361] Prefix cache hit rate: GPU: 50.00%, CPU: 0.00%
INFO 07-14 10:59:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 10:59:58 metrics.py:361] Prefix cache hit rate: GPU: 50.00%, CPU: 0.00%
INFO 07-14 11:00:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:03 metrics.py:361] Prefix cache hit rate: GPU: 50.00%, CPU: 0.00%
INFO 07-14 11:00:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:08 metrics.py:361] Prefix cache hit rate: GPU: 50.00%, CPU: 0.00%
INFO 07-14 11:00:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:13 metrics.py:361] Prefix cache hit rate: GPU: 50.00%, CPU: 0.00%
INFO: 127.0.0.1:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 11:00:19 metrics.py:345] Avg prompt throughput: 6.7 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:19 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO 07-14 11:00:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:24 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO 07-14 11:00:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:29 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO 07-14 11:00:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:34 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO 07-14 11:00:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:39 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO 07-14 11:00:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:44 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO: 127.0.0.1:39344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:39346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 11:00:51 metrics.py:345] Avg prompt throughput: 12.2 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:51 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO 07-14 11:00:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:00:56 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO 07-14 11:01:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:01 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO 07-14 11:01:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:06 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO 07-14 11:01:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:11 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO 07-14 11:01:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:16 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO: 127.0.0.1:42354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:42356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 11:01:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.4 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:21 metrics.py:361] Prefix cache hit rate: GPU: 80.00%, CPU: 0.00%
INFO 07-14 11:01:27 metrics.py:345] Avg prompt throughput: 15.4 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:27 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:01:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:32 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:01:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:37 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:01:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:42 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:01:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:47 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:01:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-14 11:01:52 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:02:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:02:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:02:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:02:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:02:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:02:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:02:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:02:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:02:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:02:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:02:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:02:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:03:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:03:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:03:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:03:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:03:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:03:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:03:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:03:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:03:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:03:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:03:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:03:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:04:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:04:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:04:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:04:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:04:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:04:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:04:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:04:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:04:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:04:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:04:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:04:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:05:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:05:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:05:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:05:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:05:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:05:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:05:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:05:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:05:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:05:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:05:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:05:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:06:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:06:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:06:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:06:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:06:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:06:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:06:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:06:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:06:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:06:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:06:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:06:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:07:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:07:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:07:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:07:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:07:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:07:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:07:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:07:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:07:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:07:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:07:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:07:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:08:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:08:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:08:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:08:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:08:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:08:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:08:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:08:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:08:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:08:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:08:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:08:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:09:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:09:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:09:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:09:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:09:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:09:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:09:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:09:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:09:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:09:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:09:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:09:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:10:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:10:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:10:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:10:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:10:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:10:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:10:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:10:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:10:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:10:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:10:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:10:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:11:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:11:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:11:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:11:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:11:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:11:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:11:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:11:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:11:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:11:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:11:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:11:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:12:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:12:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:12:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:12:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:12:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:12:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:12:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:12:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:12:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:12:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:12:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:12:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:13:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:13:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:13:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:13:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:13:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:13:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:13:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:13:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:13:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:13:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:13:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:13:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:14:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:14:01 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:14:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:14:11 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:14:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:14:21 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:14:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:14:31 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:14:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:14:41 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%
INFO 07-14 11:14:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 11:14:51 metrics.py:361] Prefix cache hit rate: GPU: 85.71%, CPU: 0.00%

104
worklogs/remote_results/2026-07-14-eager-allreduce/server_exp_graph_allreduce_seq2_b8192.log Normal file
View File

@@ -0,0 +1,104 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 10:45:59 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 10:46:01.653483: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 10:46:01.708498: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 10:46:07 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 10:46:07 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=32768, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 10:46:07 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 10:46:18 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 10:46:18 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 07-14 10:46:18 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 10:46:18 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 10:46:18 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 10:46:18 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 10:46:18 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 10:46:20 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 10:46:20 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 10:46:20 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=8115) INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=8115) INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=8114) INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=8114) INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=8115) INFO 07-14 10:46:28 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=8114) INFO 07-14 10:46:28 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=8116) INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=8116) INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=8116) INFO 07-14 10:46:28 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 10:46:28 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fdf16a327a0>, local_subscribe_port=53133, remote_subscribe_port=None)
INFO 07-14 10:46:28 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=8115) INFO 07-14 10:46:28 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=8114) INFO 07-14 10:46:28 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=8116) INFO 07-14 10:46:28 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=8116) INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=8114) INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=8115) INFO 07-14 10:46:28 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=8116) INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=8114) INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=8115) INFO 07-14 10:46:28 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:41, 1.65s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:21, 1.12it/s]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:03<00:30, 1.31s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:05<00:34, 1.58s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:06<00:29, 1.39s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:08<00:32, 1.63s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:09<00:23, 1.25s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:11<00:25, 1.43s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:12<00:22, 1.30s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:14<00:24, 1.54s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:14<00:17, 1.16s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:16<00:19, 1.38s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:18<00:20, 1.55s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:20<00:20, 1.74s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:21<00:16, 1.47s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:23<00:17, 1.75s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:24<00:12, 1.38s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:26<00:12, 1.52s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:27<00:11, 1.58s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:30<00:11, 1.86s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:31<00:08, 1.63s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:32<00:05, 1.38s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:34<00:05, 1.69s/it]
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:36<00:03, 1.78s/it]
(VllmWorkerProcess pid=8116) INFO 07-14 10:47:05 model_runner.py:1076] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=8114) INFO 07-14 10:47:05 model_runner.py:1076] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=8115) INFO 07-14 10:47:05 model_runner.py:1076] Loading model weights took 16.2303 GB