dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
900ae0b1efc19e46cd313eb3ba7eb694fa596f6b
project_6/dockerrizhi.txt
root 2d5232c5d6 comp 168 docker
2026-08-07 08:43:51 +00:00

3787 lines
677 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

我们的最开始（只截取了部分，错误太多了）日志：
2026-08-07T08:26:45.627314093Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-08-07T08:26:45.627350664Z import pynvml # type: ignore[import]
2026-08-07T08:26:47.166771940Z INFO 08-07 08:26:47 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-08-07T08:26:48.657239181Z 2026-08-07 08:26:48.657168: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-08-07T08:26:48.710793117Z 2026-08-07 08:26:48.710745: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2026-08-07T08:26:48.710798808Z To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-08-07T08:26:48.741894791Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-08-07T08:26:54.117398862Z INFO 08-07 08:26:54 api_server.py:530] vLLM API server version 0.6.3
2026-08-07T08:26:54.117663353Z INFO 08-07 08:26:54 api_server.py:531] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/model', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='half', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=1, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
2026-08-07T08:26:54.134049573Z INFO 08-07 08:26:54 config.py:1670] Downcasting torch.float32 to torch.float16.
2026-08-07T08:27:04.772029418Z INFO 08-07 08:27:04 config.py:887] Defaulting to use mp for distributed inference
2026-08-07T08:27:04.772339448Z WARNING 08-07 08:27:04 arg_utils.py:963] The model has a long context length (100000). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
2026-08-07T08:27:04.772466699Z WARNING 08-07 08:27:04 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
2026-08-07T08:27:04.775198251Z INFO 08-07 08:27:04 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/model', speculative_config=None, tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
2026-08-07T08:27:05.270845986Z WARNING 08-07 08:27:05 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
2026-08-07T08:27:05.311613008Z INFO 08-07 08:27:05 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
2026-08-07T08:27:05.362472560Z INFO 08-07 08:27:05 selector.py:115] Using XFormers backend.
2026-08-07T08:27:05.804983102Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-08-07T08:27:05.804993579Z import pynvml # type: ignore[import]
2026-08-07T08:27:05.821100558Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-08-07T08:27:05.821117178Z import pynvml # type: ignore[import]
2026-08-07T08:27:05.821119437Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-08-07T08:27:05.821121878Z import pynvml # type: ignore[import]
2026-08-07T08:27:07.336019924Z INFO 08-07 08:27:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-08-07T08:27:07.343924685Z INFO 08-07 08:27:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-08-07T08:27:07.349602702Z INFO 08-07 08:27:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-08-07T08:27:08.923183763Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-08-07T08:27:08.923188692Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-08-07T08:27:08.923204313Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-08-07T08:27:14.476866220Z (VllmWorkerProcess pid=345) INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.481729878Z (VllmWorkerProcess pid=346) INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.483088386Z (VllmWorkerProcess pid=344) INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.497162083Z (VllmWorkerProcess pid=345) INFO 08-07 08:27:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
2026-08-07T08:27:14.501909522Z (VllmWorkerProcess pid=346) INFO 08-07 08:27:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
2026-08-07T08:27:14.503443561Z (VllmWorkerProcess pid=344) INFO 08-07 08:27:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
2026-08-07T08:27:14.829071062Z INFO 08-07 08:27:14 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f18a268d690>, local_subscribe_port=44159, remote_subscribe_port=None)
2026-08-07T08:27:14.863285326Z INFO 08-07 08:27:14 model_runner.py:1119] Starting to load model /model...
2026-08-07T08:27:14.863469254Z (VllmWorkerProcess pid=345) INFO 08-07 08:27:14 model_runner.py:1119] Starting to load model /model...
2026-08-07T08:27:14.863555057Z (VllmWorkerProcess pid=344) INFO 08-07 08:27:14 model_runner.py:1119] Starting to load model /model...
2026-08-07T08:27:14.863630707Z (VllmWorkerProcess pid=346) INFO 08-07 08:27:14 model_runner.py:1119] Starting to load model /model...
2026-08-07T08:27:14.905405480Z INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.909672212Z (VllmWorkerProcess pid=344) INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.909769804Z (VllmWorkerProcess pid=345) INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.909838151Z (VllmWorkerProcess pid=346) INFO 08-07 08:27:14 selector.py:115] Using XFormers backend.
2026-08-07T08:27:14.997480301Z
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
2026-08-07T08:27:16.312189167Z
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:01<00:19, 1.31s/it]
2026-08-07T08:27:17.414772678Z
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:02<00:16, 1.19s/it]
2026-08-07T08:27:17.943863295Z
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:11, 1.13it/s]
2026-08-07T08:27:19.184723162Z
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:04<00:12, 1.03s/it]
2026-08-07T08:27:20.167398294Z
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:05<00:11, 1.01s/it]
2026-08-07T08:27:21.228769907Z
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:06<00:10, 1.03s/it]
2026-08-07T08:27:22.321152523Z
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:07<00:09, 1.05s/it]
2026-08-07T08:27:23.505443055Z
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:08<00:08, 1.09s/it]
2026-08-07T08:27:24.927087007Z
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:09<00:08, 1.20s/it]
2026-08-07T08:27:26.129186864Z
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:11<00:07, 1.20s/it]
2026-08-07T08:27:27.096541945Z
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:12<00:05, 1.13s/it]
2026-08-07T08:27:28.423075292Z
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:13<00:04, 1.19s/it]
2026-08-07T08:27:29.728377839Z
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:14<00:03, 1.22s/it]
2026-08-07T08:27:30.709363811Z
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:15<00:02, 1.15s/it]
2026-08-07T08:27:31.781864162Z
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:16<00:01, 1.13s/it]
2026-08-07T08:27:32.902095577Z
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:17<00:00, 1.12s/it]
2026-08-07T08:27:32.902243326Z
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:17<00:00, 1.12s/it]
2026-08-07T08:27:32.902260117Z
2026-08-07T08:27:33.346028029Z (VllmWorkerProcess pid=345) INFO 08-07 08:27:33 model_runner.py:1130] Loading model weights took 16.2303 GB
2026-08-07T08:27:33.346536038Z (VllmWorkerProcess pid=346) INFO 08-07 08:27:33 model_runner.py:1130] Loading model weights took 16.2303 GB
2026-08-07T08:27:33.346918807Z (VllmWorkerProcess pid=344) INFO 08-07 08:27:33 model_runner.py:1130] Loading model weights took 16.2303 GB
2026-08-07T08:27:33.357010636Z INFO 08-07 08:27:33 model_runner.py:1130] Loading model weights took 16.2303 GB
2026-08-07T08:27:55.194463382Z (VllmWorkerProcess pid=344) WARNING 08-07 08:27:55 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 0 (frac=0.9998), replacing with zeros
2026-08-07T08:27:55.194572513Z (VllmWorkerProcess pid=345) WARNING 08-07 08:27:55 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 0 (frac=0.9998), replacing with zeros
2026-08-07T08:27:55.194733437Z (VllmWorkerProcess pid=346) WARNING 08-07 08:27:55 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 0 (frac=0.9998), replacing with zeros
2026-08-07T08:27:55.194804915Z WARNING 08-07 08:27:55 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 0 (frac=0.9998), replacing with zeros
2026-08-07T08:27:55.196247134Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:55 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:55.196251374Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:55 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:55.196253846Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:55 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:55.196295437Z (VllmWorkerProcess pid=344) WARNING 08-07 08:27:55 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:55.196338474Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:55 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:55.196341519Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:55 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:55.196344149Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:55 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:55.196389650Z (VllmWorkerProcess pid=345) WARNING 08-07 08:27:55 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:55.196529296Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:55 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:55.196533947Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:55 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:55.196536534Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:55 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:55.196583400Z (VllmWorkerProcess pid=346) WARNING 08-07 08:27:55 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:55.196743459Z ERROR 08-07 08:27:55 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:55.196746436Z ERROR 08-07 08:27:55 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:55.196748856Z ERROR 08-07 08:27:55 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:55.196787680Z WARNING 08-07 08:27:55 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:56.898315178Z WARNING 08-07 08:27:56 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 1 (frac=0.9997), replacing with zeros
2026-08-07T08:27:56.898466780Z (VllmWorkerProcess pid=344) WARNING 08-07 08:27:56 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 1 (frac=0.9997), replacing with zeros
2026-08-07T08:27:56.898650736Z (VllmWorkerProcess pid=346) WARNING 08-07 08:27:56 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 1 (frac=0.9997), replacing with zeros
2026-08-07T08:27:56.898655436Z (VllmWorkerProcess pid=345) WARNING 08-07 08:27:56 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 1 (frac=0.9997), replacing with zeros
2026-08-07T08:27:56.898663860Z ERROR 08-07 08:27:56 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:56.898669821Z ERROR 08-07 08:27:56 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:56.898672533Z ERROR 08-07 08:27:56 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:56.898715835Z WARNING 08-07 08:27:56 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:56.898831930Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:56 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:56.898834864Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:56 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:56.898837465Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:56 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:56.898875999Z (VllmWorkerProcess pid=344) WARNING 08-07 08:27:56 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:56.899004651Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:56 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:56.899007519Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:56 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:56.899010032Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:56 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:56.899049092Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:56 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:56.899051749Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:56 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:56.899053941Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:56 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:56.899056040Z (VllmWorkerProcess pid=346) WARNING 08-07 08:27:56 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:56.899099993Z (VllmWorkerProcess pid=345) WARNING 08-07 08:27:56 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:58.563756628Z WARNING 08-07 08:27:58 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 2 (frac=1.0000), replacing with zeros
2026-08-07T08:27:58.563839513Z (VllmWorkerProcess pid=346) WARNING 08-07 08:27:58 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 2 (frac=1.0000), replacing with zeros
2026-08-07T08:27:58.564045316Z (VllmWorkerProcess pid=345) WARNING 08-07 08:27:58 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 2 (frac=1.0000), replacing with zeros
2026-08-07T08:27:58.564068853Z ERROR 08-07 08:27:58 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:58.564074405Z ERROR 08-07 08:27:58 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:58.564076690Z ERROR 08-07 08:27:58 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:58.564109433Z WARNING 08-07 08:27:58 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:58.564114659Z (VllmWorkerProcess pid=344) WARNING 08-07 08:27:58 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 2 (frac=1.0000), replacing with zeros
2026-08-07T08:27:58.564153710Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:58 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:58.564166969Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:58 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:58.564169301Z (VllmWorkerProcess pid=346) ERROR 08-07 08:27:58 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:58.564189936Z (VllmWorkerProcess pid=346) WARNING 08-07 08:27:58 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:58.564350334Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:58 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:58.564353987Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:58 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:58.564356275Z (VllmWorkerProcess pid=345) ERROR 08-07 08:27:58 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:58.564393401Z (VllmWorkerProcess pid=345) WARNING 08-07 08:27:58 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:27:58.564416863Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:58 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:27:58.564419370Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:58 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:27:58.564421590Z (VllmWorkerProcess pid=344) ERROR 08-07 08:27:58 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:27:58.564466363Z (VllmWorkerProcess pid=344) WARNING 08-07 08:27:58 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:04.108893189Z ERROR 08-07 08:28:04 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:04.108904912Z ERROR 08-07 08:28:04 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:04.108907541Z ERROR 08-07 08:28:04 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:04.108909754Z (VllmWorkerProcess pid=346) ERROR 08-07 08:28:04 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:04.108921805Z (VllmWorkerProcess pid=346) ERROR 08-07 08:28:04 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:04.108924463Z (VllmWorkerProcess pid=346) ERROR 08-07 08:28:04 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:04.108929227Z WARNING 08-07 08:28:04 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:04.108932279Z (VllmWorkerProcess pid=346) WARNING 08-07 08:28:04 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:04.109043313Z (VllmWorkerProcess pid=345) ERROR 08-07 08:28:04 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:04.109063172Z (VllmWorkerProcess pid=345) ERROR 08-07 08:28:04 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:04.109066423Z (VllmWorkerProcess pid=345) ERROR 08-07 08:28:04 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:04.109068809Z (VllmWorkerProcess pid=344) ERROR 08-07 08:28:04 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:04.109071243Z (VllmWorkerProcess pid=344) ERROR 08-07 08:28:04 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:04.109073470Z (VllmWorkerProcess pid=344) ERROR 08-07 08:28:04 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:04.109096223Z (VllmWorkerProcess pid=345) WARNING 08-07 08:28:04 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:04.109107675Z (VllmWorkerProcess pid=344) WARNING 08-07 08:28:04 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:05.761670405Z (VllmWorkerProcess pid=346) WARNING 08-07 08:28:05 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 4 (frac=1.0000), replacing with zeros
2026-08-07T08:28:05.761710885Z WARNING 08-07 08:28:05 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 4 (frac=1.0000), replacing with zeros
2026-08-07T08:28:05.761782158Z (VllmWorkerProcess pid=344) WARNING 08-07 08:28:05 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 4 (frac=1.0000), replacing with zeros
2026-08-07T08:28:05.761979258Z (VllmWorkerProcess pid=346) ERROR 08-07 08:28:05 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:05.761990343Z (VllmWorkerProcess pid=346) ERROR 08-07 08:28:05 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:05.761992582Z (VllmWorkerProcess pid=346) ERROR 08-07 08:28:05 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:05.762034982Z (VllmWorkerProcess pid=346) WARNING 08-07 08:28:05 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:05.762047237Z (VllmWorkerProcess pid=345) WARNING 08-07 08:28:05 qwen3_5.py:445] NaN in prefill GatedDeltaNet layer 4 (frac=1.0000), replacing with zeros
2026-08-07T08:28:05.762050526Z ERROR 08-07 08:28:05 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:05.762052833Z ERROR 08-07 08:28:05 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:05.762056152Z ERROR 08-07 08:28:05 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:05.762092143Z WARNING 08-07 08:28:05 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:05.762113248Z (VllmWorkerProcess pid=344) ERROR 08-07 08:28:05 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:05.762120235Z (VllmWorkerProcess pid=344) ERROR 08-07 08:28:05 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
2026-08-07T08:28:05.762123453Z (VllmWorkerProcess pid=344) ERROR 08-07 08:28:05 _custom_ops.py:58] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
2026-08-07T08:28:05.762165494Z (VllmWorkerProcess pid=344) WARNING 08-07 08:28:05 qwen3_5.py:913] FusedMoE native kernel failed (AttributeError: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'), falling back to pure PyTorch experts permanently.
2026-08-07T08:28:05.762351254Z (VllmWorkerProcess pid=345) ERROR 08-07 08:28:05 _custom_ops.py:58] Error in calling custom op topk_softmax: module 'ixformer.functions' has no attribute 'vllm_moe_topk_softmax'
2026-08-07T08:28:05.762359500Z (VllmWorkerProcess pid=345) ERROR 08-07 08:28:05 _custom_ops.py:58] Possibly you have built or installed an obsolete version of vllm.
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
别人168的docker日志：
2026-07-23T02:28:45.650554047Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-07-23T02:28:45.650599241Z import pynvml # type: ignore[import]
2026-07-23T02:28:47.200712741Z INFO 07-23 02:28:47 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-23T02:28:48.724165719Z 2026-07-23 02:28:48.724101: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-23T02:28:48.778248291Z 2026-07-23 02:28:48.778192: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2026-07-23T02:28:48.778258574Z To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-07-23T02:28:48.811495644Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-07-23T02:28:54.203054296Z INFO 07-23 02:28:54 api_server.py:530] vLLM API server version 0.6.3
2026-07-23T02:28:54.203282894Z INFO 07-23 02:28:54 api_server.py:531] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/model', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='half', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=256000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=4096, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=32768, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
2026-07-23T02:28:54.219648456Z INFO 07-23 02:28:54 config.py:1670] Downcasting torch.float32 to torch.float16.
2026-07-23T02:29:04.909319894Z WARNING 07-23 02:29:04 arg_utils.py:906] --enable-prefix-caching is currently not supported for multimodal models and has been disabled.
2026-07-23T02:29:04.909828848Z INFO 07-23 02:29:04 config.py:887] Defaulting to use mp for distributed inference
2026-07-23T02:29:04.910112374Z INFO 07-23 02:29:04 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=4096.
2026-07-23T02:29:04.910210781Z WARNING 07-23 02:29:04 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
2026-07-23T02:29:04.912872581Z INFO 07-23 02:29:04 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/model', speculative_config=None, tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=256000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
2026-07-23T02:29:05.406018107Z WARNING 07-23 02:29:05 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
2026-07-23T02:29:05.450119566Z INFO 07-23 02:29:05 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
2026-07-23T02:29:05.500592304Z INFO 07-23 02:29:05 selector.py:115] Using XFormers backend.
2026-07-23T02:29:05.941321070Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-07-23T02:29:05.941332229Z import pynvml # type: ignore[import]
2026-07-23T02:29:05.941336669Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-07-23T02:29:05.941348474Z import pynvml # type: ignore[import]
2026-07-23T02:29:05.941407399Z /usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2026-07-23T02:29:05.941412147Z import pynvml # type: ignore[import]
2026-07-23T02:29:07.485947726Z INFO 07-23 02:29:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-23T02:29:07.485955803Z INFO 07-23 02:29:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-23T02:29:07.485958100Z INFO 07-23 02:29:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-23T02:29:09.075001882Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-07-23T02:29:09.075016869Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-07-23T02:29:09.075076452Z WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2026-07-23T02:29:14.637614912Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:14 selector.py:115] Using XFormers backend.
2026-07-23T02:29:14.658248502Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
2026-07-23T02:29:14.676296663Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:14 selector.py:115] Using XFormers backend.
2026-07-23T02:29:14.678468262Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:14 selector.py:115] Using XFormers backend.
2026-07-23T02:29:14.696698975Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
2026-07-23T02:29:14.698898084Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
2026-07-23T02:29:15.025099398Z INFO 07-23 02:29:15 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f205f9677f0>, local_subscribe_port=41787, remote_subscribe_port=None)
2026-07-23T02:29:15.071554378Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:15 model_runner.py:1074] Starting to load model /model...
2026-07-23T02:29:15.071765260Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:15 model_runner.py:1074] Starting to load model /model...
2026-07-23T02:29:15.071769116Z INFO 07-23 02:29:15 model_runner.py:1074] Starting to load model /model...
2026-07-23T02:29:15.072467456Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:15 model_runner.py:1074] Starting to load model /model...
2026-07-23T02:29:15.123205314Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.123647472Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.136250183Z INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.142620520Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.267061728Z
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
2026-07-23T02:29:16.504035279Z
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:01<00:18, 1.24s/it]
2026-07-23T02:29:17.560515507Z
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:02<00:15, 1.13s/it]
2026-07-23T02:29:18.454967452Z
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:03<00:13, 1.02s/it]
2026-07-23T02:29:19.622930900Z
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:04<00:12, 1.08s/it]
2026-07-23T02:29:20.607005016Z
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:05<00:11, 1.05s/it]
2026-07-23T02:29:21.663620433Z
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:06<00:10, 1.05s/it]
2026-07-23T02:29:22.762559585Z
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:07<00:09, 1.07s/it]
2026-07-23T02:29:23.920513395Z
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:08<00:08, 1.09s/it]
2026-07-23T02:29:25.327235482Z
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:10<00:08, 1.19s/it]
2026-07-23T02:29:26.537079790Z
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:11<00:07, 1.20s/it]
2026-07-23T02:29:27.538590658Z
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:12<00:05, 1.14s/it]
2026-07-23T02:29:28.909915510Z
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:13<00:04, 1.21s/it]
2026-07-23T02:29:30.177921768Z
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:14<00:03, 1.23s/it]
2026-07-23T02:29:31.144529015Z
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:15<00:02, 1.15s/it]
2026-07-23T02:29:32.170218873Z
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:16<00:01, 1.11s/it]
2026-07-23T02:29:33.339901076Z
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:18<00:00, 1.13s/it]
2026-07-23T02:29:33.340059995Z
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:18<00:00, 1.13s/it]
2026-07-23T02:29:33.340067531Z
2026-07-23T02:29:33.791755457Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:29:33.791764238Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:29:33.792212417Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:29:33.799049642Z INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:32:25.902136731Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.903895709Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.905723523Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.907479743Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.915921136Z INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.918179567Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.919624182Z INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.921789252Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.950838770Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:25.950882505Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:25.950908225Z INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:25.950931658Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:26.045895062Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:26.045990261Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:26.046000888Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:26.046049182Z INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:27.731056211Z INFO 07-23 02:32:27 distributed_gpu_executor.py:57] # GPU blocks: 19259, # CPU blocks: 6553
2026-07-23T02:32:27.731106894Z INFO 07-23 02:32:27 distributed_gpu_executor.py:61] Maximum concurrency for 256000 tokens per request: 1.20x
2026-07-23T02:32:31.498383100Z INFO 07-23 02:32:31 serving_chat.py:107] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
2026-07-23T02:32:31.499707957Z INFO 07-23 02:32:31 serving_chat.py:129] Reasoning parser 'qwen3' enabled.
2026-07-23T02:32:31.499774658Z WARNING 07-23 02:32:31 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
2026-07-23T02:32:31.499825780Z INFO 07-23 02:32:31 launcher.py:19] Available routes are:
2026-07-23T02:32:31.499867858Z INFO 07-23 02:32:31 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
2026-07-23T02:32:31.499886381Z INFO 07-23 02:32:31 launcher.py:27] Route: /docs, Methods: HEAD, GET
2026-07-23T02:32:31.499904662Z INFO 07-23 02:32:31 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
2026-07-23T02:32:31.499945605Z INFO 07-23 02:32:31 launcher.py:27] Route: /redoc, Methods: HEAD, GET
2026-07-23T02:32:31.499951625Z INFO 07-23 02:32:31 launcher.py:27] Route: /health, Methods: GET
2026-07-23T02:32:31.499989247Z INFO 07-23 02:32:31 launcher.py:27] Route: /tokenize, Methods: POST
2026-07-23T02:32:31.499991563Z INFO 07-23 02:32:31 launcher.py:27] Route: /detokenize, Methods: POST
2026-07-23T02:32:31.500017919Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/models, Methods: GET
2026-07-23T02:32:31.500027643Z INFO 07-23 02:32:31 launcher.py:27] Route: /version, Methods: GET
2026-07-23T02:32:31.500047221Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/chat/completions, Methods: POST
2026-07-23T02:32:31.500071390Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/completions, Methods: POST
2026-07-23T02:32:31.500084135Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/embeddings, Methods: POST
2026-07-23T02:32:31.517724005Z INFO: Started server process [1]
2026-07-23T02:32:31.517775355Z INFO: Waiting for application startup.
2026-07-23T02:32:31.518104234Z INFO: Application startup complete.
2026-07-23T02:32:31.518434320Z INFO: Uvicorn running on socket ('0.0.0.0', 8000) (Press CTRL+C to quit)
2026-07-23T02:32:41.518080032Z INFO 07-23 02:32:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:32:44.976380698Z INFO: 172.17.0.1:34848 - "GET /health HTTP/1.1" 200 OK
2026-07-23T02:32:44.977131890Z INFO: 172.17.0.1:34848 - "GET /v1/models HTTP/1.1" 200 OK
2026-07-23T02:32:51.517794437Z INFO 07-23 02:32:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:01.518176592Z INFO 07-23 02:33:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:11.517753255Z INFO 07-23 02:33:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:21.518188707Z INFO 07-23 02:33:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:31.518515223Z INFO 07-23 02:33:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:41.518939596Z INFO 07-23 02:33:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:51.519425845Z INFO 07-23 02:33:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:01.519896974Z INFO 07-23 02:34:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:11.520418085Z INFO 07-23 02:34:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:21.520940369Z INFO 07-23 02:34:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:31.521337663Z INFO 07-23 02:34:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:41.521711416Z INFO 07-23 02:34:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:51.522131821Z INFO 07-23 02:34:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:01.522501314Z INFO 07-23 02:35:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:11.522857298Z INFO 07-23 02:35:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:21.523285280Z INFO 07-23 02:35:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:31.522702429Z INFO 07-23 02:35:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:41.523044729Z INFO 07-23 02:35:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:51.523462651Z INFO 07-23 02:35:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:01.523885993Z INFO 07-23 02:36:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:11.524260127Z INFO 07-23 02:36:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:21.523723862Z INFO 07-23 02:36:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:31.524139239Z INFO 07-23 02:36:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:41.523592375Z INFO 07-23 02:36:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:51.523995257Z INFO 07-23 02:36:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:01.524428577Z INFO 07-23 02:37:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:11.525008725Z INFO 07-23 02:37:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:21.524752003Z INFO 07-23 02:37:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:31.525171202Z INFO 07-23 02:37:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:35.774341598Z INFO: 172.28.22.23:22372 - "GET /get_model_info HTTP/1.1" 404 Not Found
2026-07-23T02:37:35.776361569Z INFO: 172.28.22.23:43360 - "GET /get_server_info HTTP/1.1" 404 Not Found
2026-07-23T02:37:35.777756478Z INFO: 172.28.22.23:30571 - "GET /version HTTP/1.1" 200 OK
2026-07-23T02:37:36.257909619Z /usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2026-07-23T02:37:36.257934560Z from pkg_resources import resource_string
2026-07-23T02:37:37.889916076Z (VllmWorkerProcess pid=346) INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.889927964Z INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.889989104Z (VllmWorkerProcess pid=344) INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.889993203Z (VllmWorkerProcess pid=345) INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.970977734Z INFO 07-23 02:37:37 metrics.py:345] Avg prompt throughput: 2.6 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:37.977175350Z (VllmWorkerProcess pid=346) INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977199535Z (VllmWorkerProcess pid=344) INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977216621Z INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977220191Z (VllmWorkerProcess pid=345) INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977862426Z (VllmWorkerProcess pid=346) INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:37.977866130Z (VllmWorkerProcess pid=345) INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:37.977868711Z INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:37.977871546Z (VllmWorkerProcess pid=344) INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:43.000160753Z INFO 07-23 02:37:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:44.269161765Z INFO: 172.28.22.23:13116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:37:44.396724776Z INFO: 172.28.22.23:7057 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:37:48.010705708Z INFO 07-23 02:37:48 metrics.py:345] Avg prompt throughput: 57.9 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:49.136976286Z INFO: 172.28.22.23:45259 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:37:53.025290553Z INFO 07-23 02:37:53 metrics.py:345] Avg prompt throughput: 7.2 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:58.061514748Z INFO 07-23 02:37:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:03.073788209Z INFO 07-23 02:38:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:06.918087930Z INFO: 172.28.22.23:34227 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:08.377827090Z INFO 07-23 02:38:08 metrics.py:345] Avg prompt throughput: 4.3 tokens/s, Avg generation throughput: 15.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:13.046591181Z INFO: 172.28.22.23:29692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:13.172739732Z INFO: 172.28.22.23:48540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:13.710792259Z INFO 07-23 02:38:13 metrics.py:345] Avg prompt throughput: 58.3 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:14.666143261Z INFO: 172.28.22.23:34379 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:18.720975352Z INFO 07-23 02:38:18 metrics.py:345] Avg prompt throughput: 74.2 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:23.725539749Z INFO 07-23 02:38:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:28.760348109Z INFO 07-23 02:38:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:33.763160416Z INFO 07-23 02:38:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:38.786603579Z INFO 07-23 02:38:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:43.822945213Z INFO 07-23 02:38:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:48.829240975Z INFO 07-23 02:38:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:53.861865865Z INFO 07-23 02:38:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:58.900819345Z INFO 07-23 02:38:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:03.930669117Z INFO 07-23 02:39:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:08.932255487Z INFO 07-23 02:39:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:13.947463156Z INFO 07-23 02:39:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:17.129519291Z INFO: 172.28.22.23:47285 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:21.526611923Z INFO 07-23 02:39:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:22.789790337Z ERROR 07-23 02:39:22 serving_chat.py:227] Error in loading multi-modal data
2026-07-23T02:39:22.789821427Z ERROR 07-23 02:39:22 serving_chat.py:227] Traceback (most recent call last):
2026-07-23T02:39:22.789824505Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 770, in _request
2026-07-23T02:39:22.789827021Z ERROR 07-23 02:39:22 serving_chat.py:227] resp = await handler(req)
2026-07-23T02:39:22.789829231Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
2026-07-23T02:39:22.789831554Z ERROR 07-23 02:39:22 serving_chat.py:227] conn = await self._connector.connect(
2026-07-23T02:39:22.789833480Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 642, in connect
2026-07-23T02:39:22.789835993Z ERROR 07-23 02:39:22 serving_chat.py:227] proto = await self._create_connection(req, traces, timeout)
2026-07-23T02:39:22.789838507Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1209, in _create_connection
2026-07-23T02:39:22.789840743Z ERROR 07-23 02:39:22 serving_chat.py:227] _, proto = await self._create_direct_connection(req, traces, timeout)
2026-07-23T02:39:22.789842704Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1550, in _create_direct_connection
2026-07-23T02:39:22.789844833Z ERROR 07-23 02:39:22 serving_chat.py:227] transp, proto = await self._wrap_create_connection(
2026-07-23T02:39:22.789857869Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1283, in _wrap_create_connection
2026-07-23T02:39:22.789859986Z ERROR 07-23 02:39:22 serving_chat.py:227] return await self._loop.create_connection(*args, **kwargs, sock=sock)
2026-07-23T02:39:22.789861916Z ERROR 07-23 02:39:22 serving_chat.py:227] File "uvloop/loop.pyx", line 2088, in create_connection
2026-07-23T02:39:22.789864184Z ERROR 07-23 02:39:22 serving_chat.py:227] File "uvloop/loop.pyx", line 2083, in uvloop.loop.Loop.create_connection
2026-07-23T02:39:22.789866163Z ERROR 07-23 02:39:22 serving_chat.py:227] asyncio.exceptions.CancelledError
2026-07-23T02:39:22.789867986Z ERROR 07-23 02:39:22 serving_chat.py:227]
2026-07-23T02:39:22.789878123Z ERROR 07-23 02:39:22 serving_chat.py:227] The above exception was the direct cause of the following exception:
2026-07-23T02:39:22.789887569Z ERROR 07-23 02:39:22 serving_chat.py:227]
2026-07-23T02:39:22.789890593Z ERROR 07-23 02:39:22 serving_chat.py:227] Traceback (most recent call last):
2026-07-23T02:39:22.789892686Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 225, in create_chat_completion
2026-07-23T02:39:22.789894923Z ERROR 07-23 02:39:22 serving_chat.py:227] mm_data = await mm_data_future
2026-07-23T02:39:22.789896983Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 248, in all_mm_data
2026-07-23T02:39:22.789899260Z ERROR 07-23 02:39:22 serving_chat.py:227] items = await asyncio.gather(*self._items)
2026-07-23T02:39:22.789901151Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/multimodal/utils.py", line 140, in async_get_and_parse_image
2026-07-23T02:39:22.789903256Z ERROR 07-23 02:39:22 serving_chat.py:227] image = await async_fetch_image(image_url)
2026-07-23T02:39:22.789905277Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/multimodal/utils.py", line 62, in async_fetch_image
2026-07-23T02:39:22.789907335Z ERROR 07-23 02:39:22 serving_chat.py:227] image_raw = await global_http_connection.async_get_bytes(
2026-07-23T02:39:22.789909226Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/connections.py", line 92, in async_get_bytes
2026-07-23T02:39:22.789911211Z ERROR 07-23 02:39:22 serving_chat.py:227] async with await self.get_async_response(url, timeout=timeout) as r:
2026-07-23T02:39:22.789913906Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1488, in __aenter__
2026-07-23T02:39:22.789915849Z ERROR 07-23 02:39:22 serving_chat.py:227] self._resp: _RetType = await self._coro
2026-07-23T02:39:22.789918323Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 622, in _request
2026-07-23T02:39:22.789921398Z ERROR 07-23 02:39:22 serving_chat.py:227] with timer:
2026-07-23T02:39:22.789929127Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/helpers.py", line 685, in __exit__
2026-07-23T02:39:22.789931189Z ERROR 07-23 02:39:22 serving_chat.py:227] raise asyncio.TimeoutError from exc_val
2026-07-23T02:39:22.789933347Z ERROR 07-23 02:39:22 serving_chat.py:227] asyncio.exceptions.TimeoutError
2026-07-23T02:39:22.789982206Z INFO: 172.28.22.23:63905 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:39:23.339790176Z INFO: 172.28.22.23:48915 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:23.862218623Z INFO: 172.28.22.23:39188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:26.537243563Z INFO 07-23 02:39:26 metrics.py:345] Avg prompt throughput: 12.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:31.709534191Z INFO: 172.28.22.23:30519 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:31.709975900Z INFO 07-23 02:39:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:32.540087790Z INFO: 172.28.22.23:28182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:36.711389406Z INFO 07-23 02:39:36 metrics.py:345] Avg prompt throughput: 6.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:38.834910565Z INFO: 172.28.22.23:25497 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:40.106985905Z INFO: 172.28.22.23:31187 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:41.375383434Z INFO: 172.28.22.23:1054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:41.736972408Z INFO 07-23 02:39:41 metrics.py:345] Avg prompt throughput: 6.6 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:42.644170272Z INFO: 172.28.22.23:10283 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:43.909888974Z INFO: 172.28.22.23:43195 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:44.026597768Z INFO: 172.28.22.23:31826 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:39:45.279682681Z INFO: 172.28.22.23:31890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:46.532238457Z INFO: 172.28.22.23:52470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:46.768052149Z INFO 07-23 02:39:46 metrics.py:345] Avg prompt throughput: 8.7 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:47.779232275Z INFO: 172.28.22.23:1935 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:47.894850660Z INFO: 172.28.22.23:24799 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:39:49.140115276Z INFO: 172.28.22.23:32610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:50.385833050Z INFO: 172.28.22.23:34946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:51.632711881Z INFO: 172.28.22.23:29640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:51.824060918Z INFO 07-23 02:39:51 metrics.py:345] Avg prompt throughput: 8.7 tokens/s, Avg generation throughput: 15.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:52.883677883Z INFO: 172.28.22.23:25348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:54.134965370Z INFO: 172.28.22.23:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:55.381846551Z INFO: 172.28.22.23:9523 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:56.629565470Z INFO: 172.28.22.23:56394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:56.827523945Z INFO 07-23 02:39:56 metrics.py:345] Avg prompt throughput: 8.8 tokens/s, Avg generation throughput: 16.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:58.129748260Z INFO: 172.28.22.23:9093 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:01.848026951Z INFO 07-23 02:40:01 metrics.py:345] Avg prompt throughput: 2.8 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:06.858910882Z INFO 07-23 02:40:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:11.862489179Z INFO 07-23 02:40:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:16.899163580Z INFO 07-23 02:40:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:21.910058417Z INFO 07-23 02:40:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:26.929209213Z INFO 07-23 02:40:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:31.962337057Z INFO 07-23 02:40:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:36.964040774Z INFO 07-23 02:40:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:41.993035873Z INFO 07-23 02:40:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:43.624005510Z INFO: 172.28.22.23:2587 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:44.061237847Z INFO: 172.28.22.23:6084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:47.185504184Z INFO: 172.28.22.23:50569 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:47.377278073Z INFO 07-23 02:40:47 metrics.py:345] Avg prompt throughput: 7.8 tokens/s, Avg generation throughput: 17.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:52.412861124Z INFO 07-23 02:40:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:57.415707222Z INFO 07-23 02:40:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:02.437584264Z INFO 07-23 02:41:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:07.476637249Z INFO 07-23 02:41:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:12.517176365Z INFO 07-23 02:41:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:17.531413276Z INFO 07-23 02:41:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:22.564441104Z INFO 07-23 02:41:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:27.577151702Z INFO 07-23 02:41:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:32.611584807Z INFO 07-23 02:41:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:37.623284038Z INFO 07-23 02:41:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:42.639031894Z INFO 07-23 02:41:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:47.641862259Z INFO 07-23 02:41:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:52.683325904Z INFO 07-23 02:41:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:57.720443718Z INFO 07-23 02:41:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:02.748390108Z INFO 07-23 02:42:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:07.784213915Z INFO 07-23 02:42:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:12.827245969Z INFO 07-23 02:42:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:17.859544046Z INFO 07-23 02:42:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:22.889429150Z INFO 07-23 02:42:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:27.923212049Z INFO 07-23 02:42:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:32.925156292Z INFO 07-23 02:42:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:37.955994920Z INFO 07-23 02:42:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:40.293610439Z INFO: 172.28.22.23:17026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:40.411301190Z INFO: 172.28.22.23:47159 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:42:40.524865477Z INFO: 172.28.22.23:53783 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:42:40.637743018Z INFO: 172.28.22.23:24159 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:42:41.181611635Z INFO: 172.28.22.23:38467 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:41.727889239Z INFO: 172.28.22.23:24963 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:42.962296089Z INFO 07-23 02:42:42 metrics.py:345] Avg prompt throughput: 64.9 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:43.783197309Z INFO: 172.28.22.23:14284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:44.459669799Z INFO: 172.28.22.23:49964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:44.575803355Z INFO: 172.28.22.23:54360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:47.973780549Z INFO 07-23 02:42:47 metrics.py:345] Avg prompt throughput: 15.0 tokens/s, Avg generation throughput: 18.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:01.858799064Z INFO 07-23 02:43:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:10.553432764Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:04<00:09, 4.87s/it]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:04<00:00, 1.31s/it]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:05<00:00, 1.69s/it]
2026-07-23T02:43:10.569991401Z INFO 07-23 02:43:10 metrics.py:345] Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:11.990491527Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.23it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.17it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 6.98it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.52it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.10it/s]
2026-07-23T02:43:13.445065774Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.22it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.22it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.07it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.63it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.14it/s]
2026-07-23T02:43:14.876448068Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.23it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.28it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.15it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.73it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.23it/s]
2026-07-23T02:43:15.701717222Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.19it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.19it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.69it/s]
2026-07-23T02:43:15.720137025Z INFO 07-23 02:43:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:16.473274583Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.21it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 5.71it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.28it/s]
2026-07-23T02:43:18.904638207Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.14it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.13it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 6.94it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.55it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 11.88it/s]
Compiling FSM index for all state transitions: 48%|████▊ | 11/23 [00:01<00:00, 13.84it/s]
Compiling FSM index for all state transitions: 57%|█████▋ | 13/23 [00:01<00:00, 15.41it/s]
Compiling FSM index for all state transitions: 65%|██████▌ | 15/23 [00:01<00:00, 16.62it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.54it/s]
Compiling FSM index for all state transitions: 87%|████████▋ | 20/23 [00:01<00:00, 18.47it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:01<00:00, 19.06it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.89it/s]
2026-07-23T02:43:21.350197196Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.14it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.24it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 7.05it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.67it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 11.99it/s]
Compiling FSM index for all state transitions: 52%|█████▏ | 12/23 [00:01<00:00, 14.60it/s]
Compiling FSM index for all state transitions: 61%|██████ | 14/23 [00:01<00:00, 15.87it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.26it/s]
Compiling FSM index for all state transitions: 87%|████████▋ | 20/23 [00:01<00:00, 18.16it/s]
Compiling FSM index for all state transitions: 96%|█████████▌| 22/23 [00:01<00:00, 18.40it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.89it/s]
2026-07-23T02:43:21.397139672Z INFO 07-23 02:43:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:22.261063391Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.22it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.24it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.73it/s]
2026-07-23T02:43:22.957955678Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.31it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.77it/s]
2026-07-23T02:43:24.351851677Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.25it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.32it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.21it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.80it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.26it/s]
2026-07-23T02:43:25.800354211Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.16it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.28it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.15it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.73it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.16it/s]
2026-07-23T02:43:26.709113498Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.23it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.73it/s]
2026-07-23T02:43:26.727308641Z INFO 07-23 02:43:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:27.404963736Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.30it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.78it/s]
2026-07-23T02:43:29.815689817Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.14it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.21it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 7.04it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.64it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 11.95it/s]
Compiling FSM index for all state transitions: 48%|████▊ | 11/23 [00:01<00:00, 13.88it/s]
Compiling FSM index for all state transitions: 57%|█████▋ | 13/23 [00:01<00:00, 15.42it/s]
Compiling FSM index for all state transitions: 65%|██████▌ | 15/23 [00:01<00:00, 16.60it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.53it/s]
Compiling FSM index for all state transitions: 87%|████████▋ | 20/23 [00:01<00:00, 18.46it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:01<00:00, 18.81it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.91it/s]
2026-07-23T02:43:31.881017891Z INFO 07-23 02:43:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:32.258419554Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.13it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.25it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 7.10it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.72it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 12.03it/s]
Compiling FSM index for all state transitions: 48%|████▊ | 11/23 [00:01<00:00, 13.97it/s]
Compiling FSM index for all state transitions: 57%|█████▋ | 13/23 [00:01<00:00, 15.50it/s]
Compiling FSM index for all state transitions: 65%|██████▌ | 15/23 [00:01<00:00, 16.69it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.58it/s]
Compiling FSM index for all state transitions: 83%|████████▎ | 19/23 [00:01<00:00, 18.25it/s]
Compiling FSM index for all state transitions: 96%|█████████▌| 22/23 [00:01<00:00, 18.93it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.89it/s]
2026-07-23T02:43:32.983539758Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.24it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.29it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.76it/s]
2026-07-23T02:43:33.672113037Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.32it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.78it/s]
2026-07-23T02:43:34.368060101Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.24it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.28it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.76it/s]
2026-07-23T02:43:34.941584523Z
Compiling FSM index for all state transitions: 0%| | 0/2 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 50%|█████ | 1/2 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 50%|█████ | 1/2 [00:00<00:00, 1.96it/s]
2026-07-23T02:43:35.230436592Z INFO: 172.28.22.23:18839 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:38.310835204Z
Compiling FSM index for all state transitions: 0%| | 0/35 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 3%|▎ | 1/35 [00:00<00:16, 2.11it/s]
Compiling FSM index for all state transitions: 9%|▊ | 3/35 [00:00<00:05, 6.13it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 5/35 [00:00<00:03, 9.43it/s]
Compiling FSM index for all state transitions: 20%|██ | 7/35 [00:00<00:02, 12.06it/s]
Compiling FSM index for all state transitions: 26%|██▌ | 9/35 [00:00<00:01, 13.89it/s]
Compiling FSM index for all state transitions: 31%|███▏ | 11/35 [00:01<00:01, 15.18it/s]
Compiling FSM index for all state transitions: 37%|███▋ | 13/35 [00:01<00:02, 10.52it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 15/35 [00:01<00:01, 12.25it/s]
Compiling FSM index for all state transitions: 49%|████▊ | 17/35 [00:01<00:01, 13.80it/s]
Compiling FSM index for all state transitions: 54%|█████▍ | 19/35 [00:01<00:01, 14.75it/s]
Compiling FSM index for all state transitions: 60%|██████ | 21/35 [00:01<00:00, 15.66it/s]
Compiling FSM index for all state transitions: 66%|██████▌ | 23/35 [00:01<00:00, 16.67it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 25/35 [00:01<00:00, 17.44it/s]
Compiling FSM index for all state transitions: 77%|███████▋ | 27/35 [00:02<00:00, 17.73it/s]
Compiling FSM index for all state transitions: 83%|████████▎ | 29/35 [00:02<00:00, 17.93it/s]
Compiling FSM index for all state transitions: 89%|████████▊ | 31/35 [00:02<00:00, 18.39it/s]
Compiling FSM index for all state transitions: 94%|█████████▍| 33/35 [00:02<00:00, 18.07it/s]
Compiling FSM index for all state transitions: 100%|██████████| 35/35 [00:02<00:00, 18.50it/s]
Compiling FSM index for all state transitions: 100%|██████████| 35/35 [00:02<00:00, 11.94it/s]
2026-07-23T02:43:38.462680310Z INFO 07-23 02:43:38 metrics.py:345] Avg prompt throughput: 3.5 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:39.428082640Z INFO: 172.28.22.23:41311 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:41.354629107Z INFO: 172.28.22.23:35141 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:41.951932899Z INFO: 172.28.22.23:40937 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:42.708616569Z INFO: 172.28.22.23:3677 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:43.378056127Z INFO: 172.28.22.23:46863 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:43.593932801Z INFO 07-23 02:43:43 metrics.py:345] Avg prompt throughput: 22.6 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:44.482046916Z INFO: 172.28.22.23:46268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:51.952117180Z INFO 07-23 02:43:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:01.952434751Z INFO 07-23 02:44:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:11.953116612Z INFO 07-23 02:44:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:21.953465417Z INFO 07-23 02:44:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:31.954404947Z INFO 07-23 02:44:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:41.955602923Z INFO 07-23 02:44:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:51.956600553Z INFO 07-23 02:44:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:01.429424131Z INFO 07-23 02:45:01 metrics.py:345] Avg prompt throughput: 33.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:02.020271472Z INFO: 172.28.22.23:10993 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:02.023173066Z INFO: 172.28.22.23:34691 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:02.542033396Z INFO: 172.28.22.23:44245 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:03.031598393Z INFO: 172.28.22.23:5495 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:03.034079308Z INFO: 172.28.22.23:33361 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:03.035816648Z INFO: 172.28.22.23:1279 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:03.038855639Z ERROR 07-23 02:45:03 serving_chat.py:221] Error in applying chat template from request
2026-07-23T02:45:03.038865861Z ERROR 07-23 02:45:03 serving_chat.py:221] Traceback (most recent call last):
2026-07-23T02:45:03.038868246Z ERROR 07-23 02:45:03 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 210, in create_chat_completion
2026-07-23T02:45:03.038870615Z ERROR 07-23 02:45:03 serving_chat.py:221] prompt = apply_hf_chat_template(
2026-07-23T02:45:03.038872879Z ERROR 07-23 02:45:03 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 574, in apply_hf_chat_template
2026-07-23T02:45:03.038875232Z ERROR 07-23 02:45:03 serving_chat.py:221] return tokenizer.apply_chat_template(
2026-07-23T02:45:03.038877096Z ERROR 07-23 02:45:03 serving_chat.py:221] File "/usr/local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1623, in apply_chat_template
2026-07-23T02:45:03.038879145Z ERROR 07-23 02:45:03 serving_chat.py:221] isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
2026-07-23T02:45:03.038881206Z ERROR 07-23 02:45:03 serving_chat.py:221] IndexError: list index out of range
2026-07-23T02:45:03.039113159Z /usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py:316: RuntimeWarning: coroutine 'AsyncMultiModalItemTracker.all_mm_data' was never awaited
2026-07-23T02:45:03.039116978Z generator = await chat(raw_request).create_chat_completion(
2026-07-23T02:45:03.039119691Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback
2026-07-23T02:45:03.039249660Z INFO: 172.28.22.23:39148 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:12.020995535Z INFO 07-23 02:45:12 metrics.py:345] Avg prompt throughput: 3.6 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:18.489898901Z INFO: 172.28.22.23:47283 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:18.707020154Z INFO 07-23 02:45:18 metrics.py:345] Avg prompt throughput: 5.8 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:23.747292153Z INFO 07-23 02:45:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:28.783886861Z INFO 07-23 02:45:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:33.805881768Z INFO 07-23 02:45:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:38.828813167Z INFO 07-23 02:45:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:43.837844599Z INFO 07-23 02:45:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:48.863019622Z INFO 07-23 02:45:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:53.902955303Z INFO 07-23 02:45:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:58.919192631Z INFO 07-23 02:45:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:03.934701110Z INFO 07-23 02:46:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:08.965144483Z INFO 07-23 02:46:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:13.969498143Z INFO 07-23 02:46:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:19.000136074Z INFO 07-23 02:46:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:24.035080024Z INFO 07-23 02:46:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:29.054704172Z INFO 07-23 02:46:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:34.075751226Z INFO 07-23 02:46:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:39.109012197Z INFO 07-23 02:46:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:44.142536722Z INFO 07-23 02:46:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:49.151030740Z INFO 07-23 02:46:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:54.155952934Z INFO 07-23 02:46:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:59.168320199Z INFO 07-23 02:46:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:04.198850701Z INFO 07-23 02:47:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:09.227769526Z INFO 07-23 02:47:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:14.236849600Z INFO 07-23 02:47:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:19.250060510Z INFO 07-23 02:47:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:24.278224796Z INFO 07-23 02:47:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:29.302744374Z INFO 07-23 02:47:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:34.340169258Z INFO 07-23 02:47:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:39.373263942Z INFO 07-23 02:47:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:44.400614703Z INFO 07-23 02:47:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:49.429303585Z INFO 07-23 02:47:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:54.454631971Z INFO 07-23 02:47:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:59.476270315Z INFO 07-23 02:47:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:04.489781001Z INFO 07-23 02:48:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:09.502845316Z INFO 07-23 02:48:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:14.525253044Z INFO 07-23 02:48:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:19.535460002Z INFO 07-23 02:48:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:24.549336142Z INFO 07-23 02:48:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:29.562276558Z INFO 07-23 02:48:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:34.582693092Z INFO 07-23 02:48:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:39.606402177Z INFO 07-23 02:48:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:44.618006935Z INFO 07-23 02:48:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:49.639887326Z INFO 07-23 02:48:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:54.661553164Z INFO 07-23 02:48:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:59.691872658Z INFO 07-23 02:48:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:04.716442787Z INFO 07-23 02:49:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:09.737600900Z INFO 07-23 02:49:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:14.765495628Z INFO 07-23 02:49:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:19.785966586Z INFO 07-23 02:49:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:24.808905236Z INFO 07-23 02:49:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:29.823237965Z INFO 07-23 02:49:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:34.840213253Z INFO 07-23 02:49:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:39.860991050Z INFO 07-23 02:49:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:44.886401826Z INFO 07-23 02:49:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:49.902498154Z INFO 07-23 02:49:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:54.932337177Z INFO 07-23 02:49:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:59.972912007Z INFO 07-23 02:49:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:05.007985393Z INFO 07-23 02:50:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:10.048282976Z INFO 07-23 02:50:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:15.051923596Z INFO 07-23 02:50:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:20.093869860Z INFO 07-23 02:50:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:25.125545341Z INFO 07-23 02:50:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:30.161930016Z INFO 07-23 02:50:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:35.204345197Z INFO 07-23 02:50:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:40.208651498Z INFO 07-23 02:50:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:45.211397957Z INFO 07-23 02:50:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:50.215514007Z INFO 07-23 02:50:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:55.226202583Z INFO 07-23 02:50:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:00.234263458Z INFO 07-23 02:51:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:05.268080221Z INFO 07-23 02:51:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:10.295027960Z INFO 07-23 02:51:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:15.333710639Z INFO 07-23 02:51:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:22.023882201Z INFO 07-23 02:51:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.1 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:32.024428280Z INFO 07-23 02:51:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:42.025051809Z INFO 07-23 02:51:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:52.025669180Z INFO 07-23 02:51:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:02.026245267Z INFO 07-23 02:52:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:12.025670280Z INFO 07-23 02:52:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:22.025991502Z INFO 07-23 02:52:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:32.026096441Z INFO 07-23 02:52:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:42.026445024Z INFO 07-23 02:52:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:52.026881786Z INFO 07-23 02:52:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:02.027443741Z INFO 07-23 02:53:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:12.027857295Z INFO 07-23 02:53:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:22.021717002Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.028015831Z INFO 07-23 02:53:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:22.034636325Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.042292538Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.049601298Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.237627231Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:53:27.059387000Z INFO 07-23 02:53:27 metrics.py:345] Avg prompt throughput: 1478.3 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:32.069675497Z INFO 07-23 02:53:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:37.075811510Z INFO 07-23 02:53:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:41.384843095Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:41.531870163Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:53:42.183280911Z INFO 07-23 02:53:42 metrics.py:345] Avg prompt throughput: 179.7 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:47.184870989Z INFO 07-23 02:53:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:52.202464890Z INFO 07-23 02:53:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:57.216053216Z INFO 07-23 02:53:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:02.236765351Z INFO 07-23 02:54:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:07.245326134Z INFO 07-23 02:54:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:12.257378000Z INFO 07-23 02:54:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:17.265723592Z INFO 07-23 02:54:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:22.274426444Z INFO 07-23 02:54:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:27.288180475Z INFO 07-23 02:54:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:32.303355245Z INFO 07-23 02:54:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:37.308955789Z INFO 07-23 02:54:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:42.316230767Z INFO 07-23 02:54:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:47.330731343Z INFO 07-23 02:54:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:52.350938716Z INFO 07-23 02:54:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:57.368955403Z INFO 07-23 02:54:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:02.387972119Z INFO 07-23 02:55:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:07.403238913Z INFO 07-23 02:55:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:12.424487516Z INFO 07-23 02:55:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:17.468209611Z INFO 07-23 02:55:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:22.468838506Z INFO 07-23 02:55:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:27.495305182Z INFO 07-23 02:55:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:31.663705730Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:55:33.559722894Z INFO 07-23 02:55:33 metrics.py:345] Avg prompt throughput: 675.4 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:39.017972246Z INFO 07-23 02:55:39 metrics.py:345] Avg prompt throughput: 2251.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:45.229906291Z INFO 07-23 02:55:45 metrics.py:345] Avg prompt throughput: 1978.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:52.031344888Z INFO 07-23 02:55:52 metrics.py:345] Avg prompt throughput: 1204.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:56.488143481Z (VllmWorkerProcess pid=346) INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:56.488439965Z INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:56.488465183Z (VllmWorkerProcess pid=345) INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:56.488468721Z (VllmWorkerProcess pid=344) INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:57.074355971Z INFO 07-23 02:55:57 metrics.py:345] Avg prompt throughput: 1703.4 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:02.075744305Z INFO 07-23 02:56:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:04.960717007Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:05.133613133Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:07.092301075Z INFO 07-23 02:56:07 metrics.py:345] Avg prompt throughput: 664.6 tokens/s, Avg generation throughput: 12.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:12.126688361Z INFO 07-23 02:56:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:14.381995529Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.389331161Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.395208292Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.402701762Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.562022048Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:18.164082686Z INFO 07-23 02:56:18 metrics.py:345] Avg prompt throughput: 1356.9 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:23.187795865Z INFO 07-23 02:56:23 metrics.py:345] Avg prompt throughput: 620.6 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:28.216477560Z INFO 07-23 02:56:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:32.311129833Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:32.500362238Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:34.377488148Z INFO 07-23 02:56:34 metrics.py:345] Avg prompt throughput: 664.8 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:39.826490851Z INFO 07-23 02:56:39 metrics.py:345] Avg prompt throughput: 2255.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:44.855411293Z INFO 07-23 02:56:44 metrics.py:345] Avg prompt throughput: 1032.6 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:49.884515182Z INFO 07-23 02:56:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:54.030501607Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:55.903654883Z INFO 07-23 02:56:55 metrics.py:345] Avg prompt throughput: 680.5 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:01.492620867Z INFO 07-23 02:57:01 metrics.py:345] Avg prompt throughput: 1348.2 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:06.518263412Z INFO 07-23 02:57:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:06.756437999Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:57:12.034175081Z INFO 07-23 02:57:12 metrics.py:345] Avg prompt throughput: 1485.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:18.176599373Z INFO 07-23 02:57:18 metrics.py:345] Avg prompt throughput: 2667.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:24.913022642Z INFO 07-23 02:57:24 metrics.py:345] Avg prompt throughput: 1824.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:32.034111195Z INFO 07-23 02:57:32 metrics.py:345] Avg prompt throughput: 1150.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:42.034654469Z INFO 07-23 02:57:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:47.041274003Z INFO 07-23 02:57:47 metrics.py:345] Avg prompt throughput: 804.5 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:51.606342181Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:57:51.804362711Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:57:53.676250964Z INFO 07-23 02:57:53 metrics.py:345] Avg prompt throughput: 617.3 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:59.365803779Z INFO 07-23 02:57:59 metrics.py:345] Avg prompt throughput: 1703.3 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:04.393685506Z INFO 07-23 02:58:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:04.609577440Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:04.615677229Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:04.798268401Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:58:10.219376350Z INFO 07-23 02:58:10 metrics.py:345] Avg prompt throughput: 2109.3 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:15.231494329Z INFO 07-23 02:58:15 metrics.py:345] Avg prompt throughput: 266.8 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:16.102093564Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:16.116114529Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:16.122546146Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:16.358547897Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:58:21.798309841Z INFO 07-23 02:58:21 metrics.py:345] Avg prompt throughput: 1871.3 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:27.758005289Z INFO 07-23 02:58:27 metrics.py:345] Avg prompt throughput: 2061.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:34.479363188Z INFO 07-23 02:58:34 metrics.py:345] Avg prompt throughput: 1828.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:41.994543666Z INFO 07-23 02:58:41 metrics.py:345] Avg prompt throughput: 1635.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:46.793774434Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:58:48.658622582Z INFO 07-23 02:58:48 metrics.py:345] Avg prompt throughput: 631.3 tokens/s, Avg generation throughput: 11.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:53.685872484Z INFO 07-23 02:58:53 metrics.py:345] Avg prompt throughput: 764.6 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:58.727252608Z INFO 07-23 02:58:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:01.342048508Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:59:01.502531224Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:05.905542940Z INFO 07-23 02:59:05 metrics.py:345] Avg prompt throughput: 1007.1 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:10.916217422Z INFO 07-23 02:59:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:15.938840078Z INFO 07-23 02:59:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:16.919636112Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:22.003161588Z INFO 07-23 02:59:22 metrics.py:345] Avg prompt throughput: 1326.5 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:27.012505251Z INFO 07-23 02:59:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:31.000804350Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:32.036163466Z INFO 07-23 02:59:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:37.075964266Z INFO 07-23 02:59:37 metrics.py:345] Avg prompt throughput: 1428.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:42.093095363Z INFO 07-23 02:59:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:44.048812647Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:47.672900564Z INFO 07-23 02:59:47 metrics.py:345] Avg prompt throughput: 1468.2 tokens/s, Avg generation throughput: 6.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:53.382936907Z INFO 07-23 02:59:53 metrics.py:345] Avg prompt throughput: 2152.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:59.847651377Z INFO 07-23 02:59:59 metrics.py:345] Avg prompt throughput: 1900.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:12.037275698Z INFO 07-23 03:00:12 metrics.py:345] Avg prompt throughput: 336.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:17.039468718Z INFO 07-23 03:00:17 metrics.py:345] Avg prompt throughput: 728.1 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:20.892612495Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:00:22.757922531Z INFO 07-23 03:00:22 metrics.py:345] Avg prompt throughput: 716.3 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:27.762148011Z INFO 07-23 03:00:27 metrics.py:345] Avg prompt throughput: 990.8 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:32.765102431Z INFO 07-23 03:00:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:37.783726602Z INFO 07-23 03:00:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:41.716682007Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:00:43.588916994Z INFO 07-23 03:00:43 metrics.py:345] Avg prompt throughput: 705.6 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:48.604975727Z INFO 07-23 03:00:48 metrics.py:345] Avg prompt throughput: 877.8 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:53.636064513Z INFO 07-23 03:00:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:58.643811919Z INFO 07-23 03:00:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:03.437865787Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:01:04.011675373Z INFO 07-23 03:01:04 metrics.py:345] Avg prompt throughput: 134.5 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:09.032151825Z INFO 07-23 03:01:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:14.076641438Z INFO 07-23 03:01:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:19.116791927Z INFO 07-23 03:01:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:24.132816105Z INFO 07-23 03:01:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:29.152510624Z INFO 07-23 03:01:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:34.164408415Z INFO 07-23 03:01:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:39.177405529Z INFO 07-23 03:01:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:44.196112311Z INFO 07-23 03:01:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:49.218177392Z INFO 07-23 03:01:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:54.260843117Z INFO 07-23 03:01:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:59.263581709Z INFO 07-23 03:01:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:04.286227306Z INFO 07-23 03:02:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:09.307925835Z INFO 07-23 03:02:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:14.334136745Z INFO 07-23 03:02:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:19.371566476Z INFO 07-23 03:02:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:24.403118388Z INFO 07-23 03:02:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:29.442979636Z INFO 07-23 03:02:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:34.460345650Z INFO 07-23 03:02:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:34.943043684Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:02:39.543923430Z INFO 07-23 03:02:39 metrics.py:345] Avg prompt throughput: 1440.3 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:43.567655344Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:02:43.733772678Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:02:45.627591234Z INFO 07-23 03:02:45 metrics.py:345] Avg prompt throughput: 673.3 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:50.087651530Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:02:50.253227675Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:02:52.041399788Z INFO 07-23 03:02:52 metrics.py:345] Avg prompt throughput: 556.3 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:57.051443405Z INFO 07-23 03:02:57 metrics.py:345] Avg prompt throughput: 1924.7 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:02.093327354Z INFO 07-23 03:03:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:07.102889862Z INFO 07-23 03:03:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:07.595301247Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:12.607251038Z INFO 07-23 03:03:12 metrics.py:345] Avg prompt throughput: 1430.3 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:14.974811119Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:20.025072743Z INFO 07-23 03:03:20 metrics.py:345] Avg prompt throughput: 1075.1 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:22.619326919Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:26.117782285Z INFO 07-23 03:03:26 metrics.py:345] Avg prompt throughput: 1344.6 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:31.802449723Z INFO 07-23 03:03:31 metrics.py:345] Avg prompt throughput: 2161.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:40.400462617Z INFO 07-23 03:03:40 metrics.py:345] Avg prompt throughput: 786.3 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:40.860411102Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:46.589024290Z INFO 07-23 03:03:46 metrics.py:345] Avg prompt throughput: 1610.1 tokens/s, Avg generation throughput: 1.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:51.589997032Z INFO 07-23 03:03:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:56.633889073Z INFO 07-23 03:03:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:57.579527268Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:03:57.746631502Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:04:02.045624284Z INFO 07-23 03:04:02 metrics.py:345] Avg prompt throughput: 756.9 tokens/s, Avg generation throughput: 3.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:04.802148446Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:04:05.027520323Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:04:08.674733439Z INFO 07-23 03:04:08 metrics.py:345] Avg prompt throughput: 1849.3 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:14.390439518Z INFO 07-23 03:04:14 metrics.py:345] Avg prompt throughput: 2149.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:20.860969569Z INFO 07-23 03:04:20 metrics.py:345] Avg prompt throughput: 1899.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:28.091096635Z INFO 07-23 03:04:28 metrics.py:345] Avg prompt throughput: 1699.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:33.666207138Z INFO 07-23 03:04:33 metrics.py:345] Avg prompt throughput: 832.6 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:37.651979949Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:04:38.689261053Z INFO 07-23 03:04:38 metrics.py:345] Avg prompt throughput: 108.1 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:43.719316648Z INFO 07-23 03:04:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:48.726194909Z INFO 07-23 03:04:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:53.773772705Z INFO 07-23 03:04:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:58.783851267Z INFO 07-23 03:04:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:03.796039763Z INFO 07-23 03:05:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:08.800370314Z INFO 07-23 03:05:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:13.817925374Z INFO 07-23 03:05:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:18.827534721Z INFO 07-23 03:05:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:23.836345906Z INFO 07-23 03:05:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:28.867665562Z INFO 07-23 03:05:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:33.879535202Z INFO 07-23 03:05:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:38.887930697Z INFO 07-23 03:05:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:43.911415061Z INFO 07-23 03:05:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:48.919604312Z INFO 07-23 03:05:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:53.923735935Z INFO 07-23 03:05:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:58.937847984Z INFO 07-23 03:05:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:03.976626873Z INFO 07-23 03:06:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:08.988814054Z INFO 07-23 03:06:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:13.996690644Z INFO 07-23 03:06:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:19.022685216Z INFO 07-23 03:06:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:24.035035366Z INFO 07-23 03:06:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:29.045998728Z INFO 07-23 03:06:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:34.065591058Z INFO 07-23 03:06:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:39.065937087Z INFO 07-23 03:06:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:44.089850754Z INFO 07-23 03:06:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:49.122224783Z INFO 07-23 03:06:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:54.140258854Z INFO 07-23 03:06:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:59.155775804Z INFO 07-23 03:06:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:04.167641219Z INFO 07-23 03:07:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:09.183289842Z INFO 07-23 03:07:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:14.190459782Z INFO 07-23 03:07:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:16.063329774Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:07:19.693310580Z INFO 07-23 03:07:19 metrics.py:345] Avg prompt throughput: 1488.7 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:24.709589906Z INFO 07-23 03:07:24 metrics.py:345] Avg prompt throughput: 289.5 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:29.752714538Z INFO 07-23 03:07:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:34.754784357Z INFO 07-23 03:07:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:37.046291229Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:07:40.663717101Z INFO 07-23 03:07:40 metrics.py:345] Avg prompt throughput: 1386.4 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:45.698396136Z INFO 07-23 03:07:45 metrics.py:345] Avg prompt throughput: 1079.9 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:50.739722502Z INFO 07-23 03:07:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:55.759442021Z INFO 07-23 03:07:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:59.292075461Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:08:00.760756817Z INFO 07-23 03:08:00 metrics.py:345] Avg prompt throughput: 68.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:05.803486926Z INFO 07-23 03:08:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:10.807800210Z INFO 07-23 03:08:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:15.810120875Z INFO 07-23 03:08:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:20.821109568Z INFO 07-23 03:08:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:25.834563489Z INFO 07-23 03:08:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:30.852701551Z INFO 07-23 03:08:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:35.895279345Z INFO 07-23 03:08:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:40.898125953Z INFO 07-23 03:08:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:45.903238274Z INFO 07-23 03:08:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:50.911176380Z INFO 07-23 03:08:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:55.948442038Z INFO 07-23 03:08:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:00.986020640Z INFO 07-23 03:09:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:05.987833861Z INFO 07-23 03:09:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:10.999189231Z INFO 07-23 03:09:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:16.010005913Z INFO 07-23 03:09:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:21.023415174Z INFO 07-23 03:09:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:26.037961024Z INFO 07-23 03:09:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:31.059518782Z INFO 07-23 03:09:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:36.076762327Z INFO 07-23 03:09:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:41.098050215Z INFO 07-23 03:09:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:46.109931334Z INFO 07-23 03:09:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:51.117802613Z INFO 07-23 03:09:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:52.198541671Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:09:58.473656203Z INFO 07-23 03:09:58 metrics.py:345] Avg prompt throughput: 1424.5 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:03.483554227Z INFO 07-23 03:10:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:08.486025664Z INFO 07-23 03:10:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:13.387287348Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:10:13.573126997Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:15.451842926Z INFO 07-23 03:10:15 metrics.py:345] Avg prompt throughput: 588.0 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:20.484742816Z INFO 07-23 03:10:20 metrics.py:345] Avg prompt throughput: 1315.2 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:25.490834189Z INFO 07-23 03:10:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:30.496012765Z INFO 07-23 03:10:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:33.890420869Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:35.776362721Z INFO 07-23 03:10:35 metrics.py:345] Avg prompt throughput: 775.7 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:40.799476081Z INFO 07-23 03:10:40 metrics.py:345] Avg prompt throughput: 834.3 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:42.302078280Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:45.904718727Z INFO 07-23 03:10:45 metrics.py:345] Avg prompt throughput: 1604.6 tokens/s, Avg generation throughput: 5.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:50.918196398Z INFO 07-23 03:10:50 metrics.py:345] Avg prompt throughput: 43.5 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:51.861459428Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:10:51.868181568Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:10:52.095460779Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:57.543735685Z INFO 07-23 03:10:57 metrics.py:345] Avg prompt throughput: 1854.6 tokens/s, Avg generation throughput: 3.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:03.506535562Z INFO 07-23 03:11:03 metrics.py:345] Avg prompt throughput: 2060.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:10.241413771Z INFO 07-23 03:11:10 metrics.py:345] Avg prompt throughput: 1824.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:17.765981052Z INFO 07-23 03:11:17 metrics.py:345] Avg prompt throughput: 1633.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:22.772423832Z INFO 07-23 03:11:22 metrics.py:345] Avg prompt throughput: 173.2 tokens/s, Avg generation throughput: 3.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:27.820717941Z INFO 07-23 03:11:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:32.868761969Z INFO 07-23 03:11:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:35.822104096Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:11:39.432629061Z INFO 07-23 03:11:39 metrics.py:345] Avg prompt throughput: 1248.1 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:42.539380381Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:11:46.145813361Z INFO 07-23 03:11:46 metrics.py:345] Avg prompt throughput: 1271.2 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:50.849891660Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:50.859074820Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:50.870425879Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:51.098867253Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:11:52.776745737Z INFO 07-23 03:11:52 metrics.py:345] Avg prompt throughput: 71.2 tokens/s, Avg generation throughput: 12.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:57.791435469Z INFO 07-23 03:11:57 metrics.py:345] Avg prompt throughput: 1751.5 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:58.518847773Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:58.688843761Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:12:04.113810544Z INFO 07-23 03:12:04 metrics.py:345] Avg prompt throughput: 1943.6 tokens/s, Avg generation throughput: 2.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:09.124640595Z INFO 07-23 03:12:09 metrics.py:345] Avg prompt throughput: 450.4 tokens/s, Avg generation throughput: 7.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:14.096840115Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:12:14.320275730Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:12:16.080406340Z INFO 07-23 03:12:16 metrics.py:345] Avg prompt throughput: 588.9 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:21.516014892Z INFO 07-23 03:12:21 metrics.py:345] Avg prompt throughput: 2260.7 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:32.515215161Z INFO 07-23 03:12:32 metrics.py:345] Avg prompt throughput: 1016.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:33.019477144Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:12:41.016003493Z INFO 07-23 03:12:41 metrics.py:345] Avg prompt throughput: 1415.5 tokens/s, Avg generation throughput: 0.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:46.059272778Z INFO 07-23 03:12:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:51.061081047Z INFO 07-23 03:12:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:56.072726396Z INFO 07-23 03:12:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:58.413592732Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:13:02.114424022Z INFO 07-23 03:13:02 metrics.py:345] Avg prompt throughput: 1355.9 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:07.138412366Z INFO 07-23 03:13:07 metrics.py:345] Avg prompt throughput: 1180.3 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:10.703528957Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:10.723774197Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:10.959621972Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:13:12.777150696Z INFO 07-23 03:13:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:18.311467651Z INFO 07-23 03:13:18 metrics.py:345] Avg prompt throughput: 2960.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:24.528764448Z INFO 07-23 03:13:24 metrics.py:345] Avg prompt throughput: 1976.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:31.514201171Z INFO 07-23 03:13:31 metrics.py:345] Avg prompt throughput: 1759.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:42.028574197Z INFO 07-23 03:13:42 metrics.py:345] Avg prompt throughput: 277.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:45.431039577Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:45.437698822Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:45.444970744Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:45.625709408Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:13:47.511514890Z INFO 07-23 03:13:47 metrics.py:345] Avg prompt throughput: 747.0 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:52.776792073Z INFO 07-23 03:13:52 metrics.py:345] Avg prompt throughput: 777.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:57.800309946Z INFO 07-23 03:13:57 metrics.py:345] Avg prompt throughput: 690.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:02.809048544Z INFO 07-23 03:14:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:06.461208755Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.468193202Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.479204061Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.490777942Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.648810696Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:14:07.835638500Z INFO 07-23 03:14:07 metrics.py:345] Avg prompt throughput: 24.3 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:12.846371069Z INFO 07-23 03:14:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:17.850632021Z INFO 07-23 03:14:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:22.856280984Z INFO 07-23 03:14:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:27.888165813Z INFO 07-23 03:14:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:32.925170980Z INFO 07-23 03:14:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:37.936372523Z INFO 07-23 03:14:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:42.944534615Z INFO 07-23 03:14:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:47.945186825Z INFO 07-23 03:14:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:52.949910632Z INFO 07-23 03:14:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:57.964030722Z INFO 07-23 03:14:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:02.982507987Z INFO 07-23 03:15:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:07.998544641Z INFO 07-23 03:15:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:13.004333260Z INFO 07-23 03:15:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:18.016318554Z INFO 07-23 03:15:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:23.029382219Z INFO 07-23 03:15:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:28.041986622Z INFO 07-23 03:15:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:33.065708355Z INFO 07-23 03:15:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:38.078323753Z INFO 07-23 03:15:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:43.091103774Z INFO 07-23 03:15:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:48.108899253Z INFO 07-23 03:15:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:53.135039995Z INFO 07-23 03:15:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:58.165410734Z INFO 07-23 03:15:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:03.183514389Z INFO 07-23 03:16:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:08.205135732Z INFO 07-23 03:16:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:13.216887456Z INFO 07-23 03:16:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:18.224956585Z INFO 07-23 03:16:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:20.572927902Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:16:20.746589986Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:16:23.267047992Z INFO 07-23 03:16:23 metrics.py:345] Avg prompt throughput: 107.7 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:28.282762875Z INFO 07-23 03:16:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:33.296712243Z INFO 07-23 03:16:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:38.326019630Z INFO 07-23 03:16:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:43.357754792Z INFO 07-23 03:16:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:48.372927578Z INFO 07-23 03:16:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:53.390454474Z INFO 07-23 03:16:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:58.399988303Z INFO 07-23 03:16:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:03.405932485Z INFO 07-23 03:17:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:08.446484958Z INFO 07-23 03:17:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:13.485751050Z INFO 07-23 03:17:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:18.523721424Z INFO 07-23 03:17:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:23.560268144Z INFO 07-23 03:17:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:28.594709480Z INFO 07-23 03:17:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:33.630996999Z INFO 07-23 03:17:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:38.670563497Z INFO 07-23 03:17:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:43.672950199Z INFO 07-23 03:17:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:48.687444828Z INFO 07-23 03:17:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:53.687871665Z INFO 07-23 03:17:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:58.699681518Z INFO 07-23 03:17:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:03.710114472Z INFO 07-23 03:18:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:08.720217881Z INFO 07-23 03:18:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:13.743844160Z INFO 07-23 03:18:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:18.764123061Z INFO 07-23 03:18:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:23.778691605Z INFO 07-23 03:18:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:28.790043878Z INFO 07-23 03:18:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:33.801253262Z INFO 07-23 03:18:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:38.807935708Z INFO 07-23 03:18:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:43.815867459Z INFO 07-23 03:18:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:48.827006448Z INFO 07-23 03:18:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:53.834181643Z INFO 07-23 03:18:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:58.865607506Z INFO 07-23 03:18:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:03.879818981Z INFO 07-23 03:19:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:08.900000692Z INFO 07-23 03:19:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:10.466463776Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:10.478993493Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:10.679321643Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:19:14.309575673Z INFO 07-23 03:19:14 metrics.py:345] Avg prompt throughput: 1514.4 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:19.322028134Z INFO 07-23 03:19:19 metrics.py:345] Avg prompt throughput: 972.8 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:24.355050263Z INFO 07-23 03:19:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:27.145073359Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:19:31.802504328Z INFO 07-23 03:19:31 metrics.py:345] Avg prompt throughput: 1008.4 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:36.308014528Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:36.316129768Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:36.507397695Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:19:38.394733836Z INFO 07-23 03:19:38 metrics.py:345] Avg prompt throughput: 621.3 tokens/s, Avg generation throughput: 15.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:43.661430418Z INFO 07-23 03:19:43 metrics.py:345] Avg prompt throughput: 1761.8 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:48.702576396Z INFO 07-23 03:19:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:53.735280536Z INFO 07-23 03:19:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:55.502354844Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:20:00.322746405Z INFO 07-23 03:20:00 metrics.py:345] Avg prompt throughput: 1168.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:02.888919108Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:20:02.897148113Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:20:03.074751698Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:20:06.698577320Z INFO 07-23 03:20:06 metrics.py:345] Avg prompt throughput: 1284.8 tokens/s, Avg generation throughput: 9.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:11.953751610Z INFO 07-23 03:20:11 metrics.py:345] Avg prompt throughput: 1225.3 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:16.958423659Z INFO 07-23 03:20:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:21.958724991Z INFO 07-23 03:20:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:26.962080264Z INFO 07-23 03:20:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:31.188353870Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:20:31.346968400Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:20:31.986096383Z INFO 07-23 03:20:31 metrics.py:345] Avg prompt throughput: 65.5 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:36.991956897Z INFO 07-23 03:20:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:42.022761244Z INFO 07-23 03:20:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:47.065410115Z INFO 07-23 03:20:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:52.071533963Z INFO 07-23 03:20:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:57.095045358Z INFO 07-23 03:20:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:02.100354460Z INFO 07-23 03:21:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:07.120462554Z INFO 07-23 03:21:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:12.144278143Z INFO 07-23 03:21:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:17.160746225Z INFO 07-23 03:21:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:22.183849444Z INFO 07-23 03:21:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:27.201981662Z INFO 07-23 03:21:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:32.222207283Z INFO 07-23 03:21:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:37.226790110Z INFO 07-23 03:21:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:42.263162787Z INFO 07-23 03:21:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:47.285217965Z INFO 07-23 03:21:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:52.294860803Z INFO 07-23 03:21:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:57.305607847Z INFO 07-23 03:21:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:02.332677933Z INFO 07-23 03:22:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:07.361450993Z INFO 07-23 03:22:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:12.386346091Z INFO 07-23 03:22:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:17.425448234Z INFO 07-23 03:22:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:22.445119521Z INFO 07-23 03:22:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:27.467709607Z INFO 07-23 03:22:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:32.488520263Z INFO 07-23 03:22:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:37.510941058Z INFO 07-23 03:22:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:42.548234171Z INFO 07-23 03:22:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:47.581833368Z INFO 07-23 03:22:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:52.623687606Z INFO 07-23 03:22:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:57.640445190Z INFO 07-23 03:22:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:02.662738298Z INFO 07-23 03:23:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:04.568763829Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:23:04.740545928Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:23:07.680005320Z INFO 07-23 03:23:07 metrics.py:345] Avg prompt throughput: 228.6 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:12.697581987Z INFO 07-23 03:23:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:17.717154268Z INFO 07-23 03:23:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:22.737304950Z INFO 07-23 03:23:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:27.757062716Z INFO 07-23 03:23:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:32.761707602Z INFO 07-23 03:23:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:37.779115679Z INFO 07-23 03:23:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:42.787757479Z INFO 07-23 03:23:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:47.821253928Z INFO 07-23 03:23:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:52.843645340Z INFO 07-23 03:23:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:57.888462280Z INFO 07-23 03:23:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:02.888897990Z INFO 07-23 03:24:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:07.905867858Z INFO 07-23 03:24:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:12.928719262Z INFO 07-23 03:24:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:17.947424275Z INFO 07-23 03:24:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:22.973047988Z INFO 07-23 03:24:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:27.993404362Z INFO 07-23 03:24:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:33.038567398Z INFO 07-23 03:24:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:38.072799979Z INFO 07-23 03:24:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:43.107134360Z INFO 07-23 03:24:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:48.139340196Z INFO 07-23 03:24:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:53.158864324Z INFO 07-23 03:24:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:58.184946563Z INFO 07-23 03:24:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:03.205227792Z INFO 07-23 03:25:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:08.227341986Z INFO 07-23 03:25:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:13.266499819Z INFO 07-23 03:25:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:18.273249978Z INFO 07-23 03:25:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:23.310387392Z INFO 07-23 03:25:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:28.347385529Z INFO 07-23 03:25:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:33.372635744Z INFO 07-23 03:25:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:38.389145564Z INFO 07-23 03:25:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:43.404210116Z INFO 07-23 03:25:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:48.431958800Z INFO 07-23 03:25:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:53.458237948Z INFO 07-23 03:25:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:58.483877167Z INFO 07-23 03:25:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:03.522336181Z INFO 07-23 03:26:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:05.090021844Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:05.095951361Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:05.295502128Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:26:08.926311950Z INFO 07-23 03:26:08 metrics.py:345] Avg prompt throughput: 1516.0 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:14.842702111Z INFO 07-23 03:26:14 metrics.py:345] Avg prompt throughput: 1164.9 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:19.852487057Z INFO 07-23 03:26:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:20.202580536Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:20.210313004Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:20.384973960Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:26:24.999260358Z INFO 07-23 03:26:24 metrics.py:345] Avg prompt throughput: 1444.2 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:30.041607883Z INFO 07-23 03:26:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:35.048754058Z INFO 07-23 03:26:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:39.100631558Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:39.109347200Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:39.304823096Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:26:41.188117195Z INFO 07-23 03:26:41 metrics.py:345] Avg prompt throughput: 667.2 tokens/s, Avg generation throughput: 15.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:50.433459245Z INFO 07-23 03:26:50 metrics.py:345] Avg prompt throughput: 1319.3 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:55.457907052Z INFO 07-23 03:26:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:00.476478203Z INFO 07-23 03:27:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:01.588214832Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:27:07.025171015Z INFO 07-23 03:27:07 metrics.py:345] Avg prompt throughput: 1876.4 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:12.629720324Z INFO 07-23 03:27:12 metrics.py:345] Avg prompt throughput: 1105.2 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:17.668027044Z INFO 07-23 03:27:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:22.673085829Z INFO 07-23 03:27:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:27.691946553Z INFO 07-23 03:27:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:29.151414399Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:27:29.743848827Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:27:29.763701013Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:27:29.917975218Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:27:32.793393663Z INFO 07-23 03:27:32 metrics.py:345] Avg prompt throughput: 802.9 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:37.836214318Z INFO 07-23 03:27:37 metrics.py:345] Avg prompt throughput: 708.7 tokens/s, Avg generation throughput: 14.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:42.859098255Z INFO 07-23 03:27:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:44.256741458Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:27:47.875184669Z INFO 07-23 03:27:47 metrics.py:345] Avg prompt throughput: 1633.2 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:52.899691834Z INFO 07-23 03:27:52 metrics.py:345] Avg prompt throughput: 155.6 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:57.919289532Z INFO 07-23 03:27:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:02.702062753Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:28:02.873611630Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:28:03.112144515Z INFO 07-23 03:28:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:08.150500129Z INFO 07-23 03:28:08 metrics.py:345] Avg prompt throughput: 1578.7 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:13.187908126Z INFO 07-23 03:28:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:17.164320870Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:28:17.327688086Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:28:19.210550904Z INFO 07-23 03:28:19 metrics.py:345] Avg prompt throughput: 680.1 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:24.225292173Z INFO 07-23 03:28:24 metrics.py:345] Avg prompt throughput: 800.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:29.226764995Z INFO 07-23 03:28:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:31.869759250Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:28:34.263598007Z INFO 07-23 03:28:34 metrics.py:345] Avg prompt throughput: 132.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:39.292111567Z INFO 07-23 03:28:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:44.324452648Z INFO 07-23 03:28:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:49.348232646Z INFO 07-23 03:28:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:54.355931285Z INFO 07-23 03:28:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:59.363795090Z INFO 07-23 03:28:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:04.392516862Z INFO 07-23 03:29:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:09.415443309Z INFO 07-23 03:29:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:14.436739524Z INFO 07-23 03:29:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:19.457395216Z INFO 07-23 03:29:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:24.474718167Z INFO 07-23 03:29:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:29.501523520Z INFO 07-23 03:29:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:34.523233969Z INFO 07-23 03:29:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:39.552660459Z INFO 07-23 03:29:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:44.570080552Z INFO 07-23 03:29:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:49.596454253Z INFO 07-23 03:29:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:54.615672916Z INFO 07-23 03:29:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:59.633932098Z INFO 07-23 03:29:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:02.589700120Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:02.778803816Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:04.677928907Z INFO 07-23 03:30:04 metrics.py:345] Avg prompt throughput: 812.1 tokens/s, Avg generation throughput: 13.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:09.720978113Z INFO 07-23 03:30:09 metrics.py:345] Avg prompt throughput: 952.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:11.253121488Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:14.873610618Z INFO 07-23 03:30:14 metrics.py:345] Avg prompt throughput: 1589.9 tokens/s, Avg generation throughput: 5.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:21.905152531Z INFO 07-23 03:30:21 metrics.py:345] Avg prompt throughput: 1420.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:26.927246350Z INFO 07-23 03:30:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:26.949212854Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:26.957860631Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:26.966418697Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:27.141252545Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:31.939042290Z INFO 07-23 03:30:31 metrics.py:345] Avg prompt throughput: 1712.4 tokens/s, Avg generation throughput: 2.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:36.949470944Z INFO 07-23 03:30:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:40.533602444Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:40.691497271Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:41.010347463Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:41.018689226Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:41.178014687Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:43.067206037Z INFO 07-23 03:30:43 metrics.py:345] Avg prompt throughput: 685.9 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:48.085902193Z INFO 07-23 03:30:48 metrics.py:345] Avg prompt throughput: 617.9 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:53.113753547Z INFO 07-23 03:30:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:53.864374472Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:53.875008951Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:54.049288150Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:58.801164893Z INFO 07-23 03:30:58 metrics.py:345] Avg prompt throughput: 1346.1 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:01.135308567Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:31:06.029877137Z INFO 07-23 03:31:06 metrics.py:345] Avg prompt throughput: 1074.5 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:08.592483770Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:31:08.960823866Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:31:12.640101178Z INFO 07-23 03:31:12 metrics.py:345] Avg prompt throughput: 1239.3 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:18.383940479Z INFO 07-23 03:31:18 metrics.py:345] Avg prompt throughput: 2139.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:24.879158343Z INFO 07-23 03:31:24 metrics.py:345] Avg prompt throughput: 1891.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:32.155291335Z INFO 07-23 03:31:32 metrics.py:345] Avg prompt throughput: 1688.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:37.443808749Z INFO 07-23 03:31:37 metrics.py:345] Avg prompt throughput: 1549.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:43.111724266Z INFO 07-23 03:31:43 metrics.py:345] Avg prompt throughput: 1445.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:49.164283827Z INFO 07-23 03:31:49 metrics.py:345] Avg prompt throughput: 1353.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:55.618757782Z INFO 07-23 03:31:55 metrics.py:345] Avg prompt throughput: 1269.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:02.450252476Z INFO 07-23 03:32:02 metrics.py:345] Avg prompt throughput: 1199.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:09.683638726Z INFO 07-23 03:32:09 metrics.py:345] Avg prompt throughput: 1132.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:17.325293296Z INFO 07-23 03:32:17 metrics.py:345] Avg prompt throughput: 1072.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:23.119540943Z INFO 07-23 03:32:23 metrics.py:345] Avg prompt throughput: 706.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:33.119590916Z INFO 07-23 03:32:33 metrics.py:345] Avg prompt throughput: 409.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:43.120993751Z INFO 07-23 03:32:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:51.139668086Z INFO 07-23 03:32:51 metrics.py:345] Avg prompt throughput: 349.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:56.182102278Z INFO 07-23 03:32:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:01.222431844Z INFO 07-23 03:33:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:06.255444585Z INFO 07-23 03:33:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:11.294163817Z INFO 07-23 03:33:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:16.330648490Z INFO 07-23 03:33:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:21.366805857Z INFO 07-23 03:33:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:26.401669203Z INFO 07-23 03:33:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:31.427649231Z INFO 07-23 03:33:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:36.462316305Z INFO 07-23 03:33:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:41.488932088Z INFO 07-23 03:33:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:46.518394807Z INFO 07-23 03:33:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:51.547857990Z INFO 07-23 03:33:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:56.580255579Z INFO 07-23 03:33:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:01.580987774Z INFO 07-23 03:34:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:06.616073565Z INFO 07-23 03:34:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:11.650597177Z INFO 07-23 03:34:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:16.654305379Z INFO 07-23 03:34:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:21.683189679Z INFO 07-23 03:34:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:26.707783194Z INFO 07-23 03:34:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:31.739851567Z INFO 07-23 03:34:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:36.764364675Z INFO 07-23 03:34:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:41.785282509Z INFO 07-23 03:34:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:46.807385509Z INFO 07-23 03:34:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:51.832113797Z INFO 07-23 03:34:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:56.852758637Z INFO 07-23 03:34:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:01.877195030Z INFO 07-23 03:35:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:06.909124996Z INFO 07-23 03:35:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:11.936863441Z INFO 07-23 03:35:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:16.969294745Z INFO 07-23 03:35:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:22.001685150Z INFO 07-23 03:35:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:27.035357579Z INFO 07-23 03:35:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:32.070780441Z INFO 07-23 03:35:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:37.109800889Z INFO 07-23 03:35:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:42.144817317Z INFO 07-23 03:35:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:47.182019980Z INFO 07-23 03:35:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:52.225007586Z INFO 07-23 03:35:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:57.259046506Z INFO 07-23 03:35:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:02.297575559Z INFO 07-23 03:36:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:07.340493043Z INFO 07-23 03:36:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:12.378520548Z INFO 07-23 03:36:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:17.409447662Z INFO 07-23 03:36:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:22.439119517Z INFO 07-23 03:36:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:27.442704506Z INFO 07-23 03:36:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:32.480948868Z INFO 07-23 03:36:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:37.529398347Z INFO 07-23 03:36:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:42.574099184Z INFO 07-23 03:36:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:47.612483559Z INFO 07-23 03:36:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:52.652951855Z INFO 07-23 03:36:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:57.687904692Z INFO 07-23 03:36:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:02.726593377Z INFO 07-23 03:37:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:07.730913728Z INFO 07-23 03:37:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:12.774245347Z INFO 07-23 03:37:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:17.813872698Z INFO 07-23 03:37:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:22.848066921Z INFO 07-23 03:37:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:27.882473147Z INFO 07-23 03:37:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:32.917401272Z INFO 07-23 03:37:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:37.932243776Z INFO 07-23 03:37:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:42.963779527Z INFO 07-23 03:37:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:48.007317544Z INFO 07-23 03:37:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:53.048938883Z INFO 07-23 03:37:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:58.085090714Z INFO 07-23 03:37:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:59.337134045Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:03.126573524Z INFO 07-23 03:38:03 metrics.py:345] Avg prompt throughput: 1624.9 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:07.915203230Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:38:07.925057068Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:38:08.094963074Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:09.982564809Z INFO 07-23 03:38:09 metrics.py:345] Avg prompt throughput: 715.7 tokens/s, Avg generation throughput: 12.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:14.987905774Z INFO 07-23 03:38:14 metrics.py:345] Avg prompt throughput: 1116.8 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:19.993703211Z INFO 07-23 03:38:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:23.734983446Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:25.623769661Z INFO 07-23 03:38:25 metrics.py:345] Avg prompt throughput: 727.5 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:30.635619226Z INFO 07-23 03:38:30 metrics.py:345] Avg prompt throughput: 1021.6 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:34.343758361Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:38:34.566766277Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:36.464600680Z INFO 07-23 03:38:36 metrics.py:345] Avg prompt throughput: 702.7 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:41.913518197Z INFO 07-23 03:38:41 metrics.py:345] Avg prompt throughput: 2255.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:53.127802320Z INFO 07-23 03:38:53 metrics.py:345] Avg prompt throughput: 730.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:57.626063560Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:59.523823531Z INFO 07-23 03:38:59 metrics.py:345] Avg prompt throughput: 1128.4 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:04.973947484Z INFO 07-23 03:39:04 metrics.py:345] Avg prompt throughput: 2254.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:13.127817976Z INFO 07-23 03:39:13 metrics.py:345] Avg prompt throughput: 502.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:18.136657170Z INFO 07-23 03:39:18 metrics.py:345] Avg prompt throughput: 697.6 tokens/s, Avg generation throughput: 16.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:19.550416132Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:39:23.976378415Z INFO 07-23 03:39:23 metrics.py:345] Avg prompt throughput: 1237.9 tokens/s, Avg generation throughput: 4.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:29.002950812Z INFO 07-23 03:39:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:34.041860533Z INFO 07-23 03:39:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:34.620637687Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:39:41.578722845Z INFO 07-23 03:39:41 metrics.py:345] Avg prompt throughput: 1490.8 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:46.612817940Z INFO 07-23 03:39:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:51.651629828Z INFO 07-23 03:39:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:56.654212752Z INFO 07-23 03:39:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:58.976469155Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:40:02.603723546Z INFO 07-23 03:40:02 metrics.py:345] Avg prompt throughput: 1376.9 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:08.305651965Z INFO 07-23 03:40:08 metrics.py:345] Avg prompt throughput: 2155.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:16.236480761Z INFO 07-23 03:40:16 metrics.py:345] Avg prompt throughput: 478.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:21.267816676Z INFO 07-23 03:40:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:26.295602231Z INFO 07-23 03:40:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:28.778018078Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:40:32.408324583Z INFO 07-23 03:40:32 metrics.py:345] Avg prompt throughput: 1340.2 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:38.108997575Z INFO 07-23 03:40:38 metrics.py:345] Avg prompt throughput: 2155.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:43.129334599Z INFO 07-23 03:40:43 metrics.py:345] Avg prompt throughput: 1631.7 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:48.160479129Z INFO 07-23 03:40:48 metrics.py:345] Avg prompt throughput: 120.6 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:53.164874777Z INFO 07-23 03:40:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:58.169636135Z INFO 07-23 03:40:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:59.795739240Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:41:03.414206853Z INFO 07-23 03:41:03 metrics.py:345] Avg prompt throughput: 1562.0 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:08.455624617Z INFO 07-23 03:41:08 metrics.py:345] Avg prompt throughput: 728.7 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:13.456923903Z INFO 07-23 03:41:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:18.485525514Z INFO 07-23 03:41:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:23.498236924Z INFO 07-23 03:41:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:26.268303762Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:41:31.070539610Z INFO 07-23 03:41:31 metrics.py:345] Avg prompt throughput: 1015.4 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:36.096941176Z INFO 07-23 03:41:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:41.130726930Z INFO 07-23 03:41:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:41.559287741Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:41:41.969406342Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:41:47.474648078Z INFO 07-23 03:41:47 metrics.py:345] Avg prompt throughput: 1937.0 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:53.131561420Z INFO 07-23 03:41:53 metrics.py:345] Avg prompt throughput: 1448.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:00.176943964Z INFO 07-23 03:42:00 metrics.py:345] Avg prompt throughput: 2325.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:07.700147555Z INFO 07-23 03:42:07 metrics.py:345] Avg prompt throughput: 1633.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:13.132480415Z INFO 07-23 03:42:13 metrics.py:345] Avg prompt throughput: 754.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:19.035572074Z INFO 07-23 03:42:19 metrics.py:345] Avg prompt throughput: 2081.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:33.133304474Z INFO 07-23 03:42:33 metrics.py:345] Avg prompt throughput: 290.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:43.134438022Z INFO 07-23 03:42:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:48.154183452Z INFO 07-23 03:42:48 metrics.py:345] Avg prompt throughput: 735.3 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:53.191208919Z INFO 07-23 03:42:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:57.644819022Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:42:59.536790168Z INFO 07-23 03:42:59 metrics.py:345] Avg prompt throughput: 645.5 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:04.984151481Z INFO 07-23 03:43:04 metrics.py:345] Avg prompt throughput: 2255.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:13.134871432Z INFO 07-23 03:43:13 metrics.py:345] Avg prompt throughput: 1005.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:18.371162432Z INFO 07-23 03:43:18 metrics.py:345] Avg prompt throughput: 757.4 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:23.343108547Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:43:23.588052243Z INFO 07-23 03:43:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:28.848071558Z INFO 07-23 03:43:28 metrics.py:345] Avg prompt throughput: 2336.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:34.822293294Z INFO 07-23 03:43:34 metrics.py:345] Avg prompt throughput: 2056.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:41.559053064Z INFO 07-23 03:43:41 metrics.py:345] Avg prompt throughput: 1824.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:42.134554199Z INFO: 172.28.22.23:54024 - "GET /v1/models HTTP/1.1" 200 OK
2026-07-23T03:43:49.093692247Z INFO 07-23 03:43:49 metrics.py:345] Avg prompt throughput: 1630.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:54.573982630Z INFO 07-23 03:43:54 metrics.py:345] Avg prompt throughput: 1494.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:00.419430557Z INFO 07-23 03:44:00 metrics.py:345] Avg prompt throughput: 1401.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:06.652667894Z INFO 07-23 03:44:06 metrics.py:345] Avg prompt throughput: 1314.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:13.273466906Z INFO 07-23 03:44:13 metrics.py:345] Avg prompt throughput: 1237.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:23.589944720Z INFO 07-23 03:44:23 metrics.py:345] Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:33.589622516Z INFO 07-23 03:44:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:40.373070879Z INFO 07-23 03:44:40 metrics.py:345] Avg prompt throughput: 477.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:45.378779262Z INFO 07-23 03:44:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:49.409870456Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:44:51.294134680Z INFO 07-23 03:44:51 metrics.py:345] Avg prompt throughput: 692.4 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:57.327566458Z INFO 07-23 03:44:57 metrics.py:345] Avg prompt throughput: 1304.6 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:02.343004242Z INFO 07-23 03:45:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:07.353037375Z INFO 07-23 03:45:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:08.703038876Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:45:08.945392988Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:45:12.589865901Z INFO 07-23 03:45:12 metrics.py:345] Avg prompt throughput: 1564.3 tokens/s, Avg generation throughput: 5.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:18.308064429Z INFO 07-23 03:45:18 metrics.py:345] Avg prompt throughput: 2148.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:23.590532919Z INFO 07-23 03:45:23 metrics.py:345] Avg prompt throughput: 1550.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:29.508225758Z INFO 07-23 03:45:29 metrics.py:345] Avg prompt throughput: 2076.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:40.164335600Z INFO 07-23 03:45:40 metrics.py:345] Avg prompt throughput: 286.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:45.215691270Z INFO 07-23 03:45:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:50.262555523Z INFO 07-23 03:45:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:55.305684193Z INFO 07-23 03:45:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:00.329990814Z INFO 07-23 03:46:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:05.358893348Z INFO 07-23 03:46:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:10.388523949Z INFO 07-23 03:46:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:15.420504664Z INFO 07-23 03:46:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:20.448707580Z INFO 07-23 03:46:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:25.464481774Z INFO 07-23 03:46:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:30.502872201Z INFO 07-23 03:46:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:35.530078168Z INFO 07-23 03:46:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:40.559825975Z INFO 07-23 03:46:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:45.595368097Z INFO 07-23 03:46:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:50.632083164Z INFO 07-23 03:46:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:55.670123461Z INFO 07-23 03:46:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:00.692064975Z INFO 07-23 03:47:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:05.731449900Z INFO 07-23 03:47:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:10.775862113Z INFO 07-23 03:47:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:15.789807502Z INFO 07-23 03:47:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:18.030344386Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:47:18.267069376Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:47:21.908553681Z INFO 07-23 03:47:21 metrics.py:345] Avg prompt throughput: 1338.8 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:27.607452205Z INFO 07-23 03:47:27 metrics.py:345] Avg prompt throughput: 2156.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:33.592167279Z INFO 07-23 03:47:33 metrics.py:345] Avg prompt throughput: 684.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:38.615627195Z INFO 07-23 03:47:38 metrics.py:345] Avg prompt throughput: 610.3 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:43.652542974Z INFO 07-23 03:47:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:48.069171000Z ERROR 07-23 03:47:48 serving_chat.py:221] Error in applying chat template from request
2026-07-23T03:47:48.069190665Z ERROR 07-23 03:47:48 serving_chat.py:221] Traceback (most recent call last):
2026-07-23T03:47:48.069193533Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 188, in create_chat_completion
2026-07-23T03:47:48.069196058Z ERROR 07-23 03:47:48 serving_chat.py:221] conversation, mm_data_future = parse_chat_messages_futures(
2026-07-23T03:47:48.069198382Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 551, in parse_chat_messages_futures
2026-07-23T03:47:48.069200645Z ERROR 07-23 03:47:48 serving_chat.py:221] sub_messages = _parse_chat_message_content(msg, mm_tracker)
2026-07-23T03:47:48.069202837Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 477, in _parse_chat_message_content
2026-07-23T03:47:48.069204889Z ERROR 07-23 03:47:48 serving_chat.py:221] result = _parse_chat_message_content_parts(
2026-07-23T03:47:48.069206940Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 429, in _parse_chat_message_content_parts
2026-07-23T03:47:48.069209541Z ERROR 07-23 03:47:48 serving_chat.py:221] mm_parser.parse_image(image_url["url"])
2026-07-23T03:47:48.069211572Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 311, in parse_image
2026-07-23T03:47:48.069213819Z ERROR 07-23 03:47:48 serving_chat.py:221] placeholder = self._tracker.add("image", image_coro)
2026-07-23T03:47:48.069215830Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 220, in add
2026-07-23T03:47:48.069218119Z ERROR 07-23 03:47:48 serving_chat.py:221] raise ValueError(
2026-07-23T03:47:48.069229290Z ERROR 07-23 03:47:48 serving_chat.py:221] ValueError: At most 1 image(s) may be provided in one request.
2026-07-23T03:47:48.069293988Z /usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py:222: RuntimeWarning: coroutine 'async_get_and_parse_image' was never awaited
2026-07-23T03:47:48.069300137Z return self.create_error_response(str(e))
2026-07-23T03:47:48.069302341Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback
2026-07-23T03:47:48.069481495Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:47:48.325722518Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:47:50.228821656Z INFO 07-23 03:47:50 metrics.py:345] Avg prompt throughput: 622.9 tokens/s, Avg generation throughput: 14.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:55.685485411Z INFO 07-23 03:47:55 metrics.py:345] Avg prompt throughput: 2251.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:01.889056918Z INFO 07-23 03:48:01 metrics.py:345] Avg prompt throughput: 1980.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:08.880652390Z INFO 07-23 03:48:08 metrics.py:345] Avg prompt throughput: 1757.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:20.950602402Z INFO 07-23 03:48:20 metrics.py:345] Avg prompt throughput: 547.1 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:25.952959584Z INFO 07-23 03:48:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:30.999396546Z INFO 07-23 03:48:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:36.044256010Z INFO 07-23 03:48:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:41.044599828Z INFO 07-23 03:48:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:46.083073696Z INFO 07-23 03:48:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:51.124521665Z INFO 07-23 03:48:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:56.168277560Z INFO 07-23 03:48:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:01.212516850Z INFO 07-23 03:49:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:06.258871180Z INFO 07-23 03:49:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:11.303607987Z INFO 07-23 03:49:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:16.352861160Z INFO 07-23 03:49:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:21.355919111Z INFO 07-23 03:49:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:23.979457774Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:49:27.605136815Z INFO 07-23 03:49:27 metrics.py:345] Avg prompt throughput: 1310.9 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:33.296245524Z INFO 07-23 03:49:33 metrics.py:345] Avg prompt throughput: 2159.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:38.656394179Z INFO 07-23 03:49:38 metrics.py:345] Avg prompt throughput: 1597.3 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:41.259500869Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:49:44.876448503Z INFO 07-23 03:49:44 metrics.py:345] Avg prompt throughput: 1317.1 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:50.779280791Z INFO 07-23 03:49:50 metrics.py:345] Avg prompt throughput: 1163.2 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:55.782459543Z INFO 07-23 03:49:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:00.821850285Z INFO 07-23 03:50:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:05.860302968Z INFO 07-23 03:50:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:10.896836932Z INFO 07-23 03:50:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:15.907192499Z INFO 07-23 03:50:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:19.131493471Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:50:21.026655055Z INFO 07-23 03:50:21 metrics.py:345] Avg prompt throughput: 800.1 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:26.482545766Z INFO 07-23 03:50:26 metrics.py:345] Avg prompt throughput: 2252.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:32.675695952Z INFO 07-23 03:50:32 metrics.py:345] Avg prompt throughput: 1984.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:37.717894719Z INFO 07-23 03:50:37 metrics.py:345] Avg prompt throughput: 104.5 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:38.747799235Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:50:43.596272691Z INFO 07-23 03:50:43 metrics.py:345] Avg prompt throughput: 1393.6 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:49.494533413Z INFO 07-23 03:50:49 metrics.py:345] Avg prompt throughput: 1317.8 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:54.495926964Z INFO 07-23 03:50:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:59.532585217Z INFO 07-23 03:50:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:51:04.243701758Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:51:04.695557111Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, Traceback (most recent call last):
2026-07-23T03:51:04.695590033Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
2026-07-23T03:51:04.695593398Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
2026-07-23T03:51:04.695597304Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695599822Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695601960Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 69, in start_worker_execution_loop
2026-07-23T03:51:04.695604422Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.execute_model(execute_model_req=None)
2026-07-23T03:51:04.695606461Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.695608857Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.model_runner.execute_model(
2026-07-23T03:51:04.695611175Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695613359Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695615304Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.695617660Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.695619640Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695621844Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695623828Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695628609Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695630599Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.695632812Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states = self.model(
2026-07-23T03:51:04.695634801Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695636954Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695639175Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695641326Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695643268Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.695645430Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = layer(
2026-07-23T03:51:04.695647355Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695649500Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695651500Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695653916Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695655924Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.695658142Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.695660265Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695662484Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695667765Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695670920Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695672956Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.695675139Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.695677985Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.695680320Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_native(x, residual)
2026-07-23T03:51:04.695684136Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.695686306Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.695688422Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 158, in forward_static
2026-07-23T03:51:04.695690647Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] variance = x.pow(2).mean(dim=-1, keepdim=True)
2026-07-23T03:51:04.695693383Z (VllmWorkerProcess pid=345) (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 1 has a total capacty of 31.72 GiB of which 74.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, Traceback (most recent call last):
2026-07-23T03:51:04.695697049Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
2026-07-23T03:51:04.695699616Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
2026-07-23T03:51:04.695701640Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695706279Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695708636Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 69, in start_worker_execution_loop
2026-07-23T03:51:04.695710796Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.execute_model(execute_model_req=None)
2026-07-23T03:51:04.695712766Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.695714957Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.model_runner.execute_model(
2026-07-23T03:51:04.695716932Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695719271Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695721336Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.695723498Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.695725472Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695727695Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695729775Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695732069Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695734030Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.695736157Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states = self.model(
2026-07-23T03:51:04.695738121Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695742683Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695744681Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695746918Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695748963Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.695751447Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = layer(
2026-07-23T03:51:04.695753660Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695755775Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695757874Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695759993Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695762002Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.695764118Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.695766114Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695768209Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695770214Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695772460Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695774624Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.695778783Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.695781237Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.695783998Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_native(x, residual)
2026-07-23T03:51:04.695785906Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.695788026Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.695790208Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 158, in forward_static
2026-07-23T03:51:04.695792433Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] variance = x.pow(2).mean(dim=-1, keepdim=True)
2026-07-23T03:51:04.695794820Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.695797659Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231]
2026-07-23T03:51:04.695799657Z ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 1 has a total capacty of 31.72 GiB of which 74.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.695802893Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231]
2026-07-23T03:51:04.699701286Z ERROR 07-23 03:51:04 async_llm_engine.py:66] Engine background task failed
2026-07-23T03:51:04.699707282Z ERROR 07-23 03:51:04 async_llm_engine.py:66] Traceback (most recent call last):
2026-07-23T03:51:04.699709690Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 56, in _log_task_completion
2026-07-23T03:51:04.699712553Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return_value = task.result()
2026-07-23T03:51:04.699722044Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 857, in run_engine_loop
2026-07-23T03:51:04.699724371Z ERROR 07-23 03:51:04 async_llm_engine.py:66] result = task.result()
2026-07-23T03:51:04.699726357Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 776, in engine_step
2026-07-23T03:51:04.699728437Z ERROR 07-23 03:51:04 async_llm_engine.py:66] request_outputs = await self.engine.step_async(virtual_engine)
2026-07-23T03:51:04.699730549Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 348, in step_async
2026-07-23T03:51:04.699732927Z ERROR 07-23 03:51:04 async_llm_engine.py:66] outputs = await self.model_executor.execute_model_async(
2026-07-23T03:51:04.699734814Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 181, in execute_model_async
2026-07-23T03:51:04.699736873Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return await self._driver_execute_model_async(execute_model_req)
2026-07-23T03:51:04.699738752Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 224, in _driver_execute_model_async
2026-07-23T03:51:04.699740932Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return await self.driver_exec_model(execute_model_req)
2026-07-23T03:51:04.699743070Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2026-07-23T03:51:04.699745042Z ERROR 07-23 03:51:04 async_llm_engine.py:66] result = self.fn(*self.args, **self.kwargs)
2026-07-23T03:51:04.699747316Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.699749564Z ERROR 07-23 03:51:04 async_llm_engine.py:66] output = self.model_runner.execute_model(
2026-07-23T03:51:04.699751539Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.699753508Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return func(*args, **kwargs)
2026-07-23T03:51:04.699755607Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.699758036Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.699760153Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699762173Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699764016Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699768920Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699770911Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.699772926Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_states = self.model(
2026-07-23T03:51:04.699774758Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699776744Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699778565Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699780536Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699782347Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.699784609Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_states, residual = layer(
2026-07-23T03:51:04.699786447Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699788829Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699790628Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699792704Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699856811Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.699859662Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.699861913Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699864092Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699866211Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699868266Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699870390Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.699872478Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.699876920Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.699879423Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self.forward_native(x, residual)
2026-07-23T03:51:04.699881751Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.699884537Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.699886523Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_static
2026-07-23T03:51:04.699888605Z ERROR 07-23 03:51:04 async_llm_engine.py:66] x = x.float()
2026-07-23T03:51:04.699890682Z ERROR 07-23 03:51:04 async_llm_engine.py:66] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.700138219Z Exception in callback functools.partial(<function _log_task_completion at 0x7f206fedf1c0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f206034f010>>)
2026-07-23T03:51:04.700142915Z handle: <Handle functools.partial(<function _log_task_completion at 0x7f206fedf1c0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f206034f010>>)>
2026-07-23T03:51:04.700146529Z Traceback (most recent call last):
2026-07-23T03:51:04.700148465Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 56, in _log_task_completion
2026-07-23T03:51:04.700150727Z return_value = task.result()
2026-07-23T03:51:04.700152569Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 857, in run_engine_loop
2026-07-23T03:51:04.700154521Z result = task.result()
2026-07-23T03:51:04.700156285Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 776, in engine_step
2026-07-23T03:51:04.700158380Z request_outputs = await self.engine.step_async(virtual_engine)
2026-07-23T03:51:04.700160278Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 348, in step_async
2026-07-23T03:51:04.700162359Z outputs = await self.model_executor.execute_model_async(
2026-07-23T03:51:04.700164171Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 181, in execute_model_async
2026-07-23T03:51:04.700166399Z return await self._driver_execute_model_async(execute_model_req)
2026-07-23T03:51:04.700170872Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 224, in _driver_execute_model_async
2026-07-23T03:51:04.700172919Z return await self.driver_exec_model(execute_model_req)
2026-07-23T03:51:04.700174704Z File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2026-07-23T03:51:04.700176814Z result = self.fn(*self.args, **self.kwargs)
2026-07-23T03:51:04.700178633Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.700180563Z output = self.model_runner.execute_model(
2026-07-23T03:51:04.700182330Z File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.700184286Z return func(*args, **kwargs)
2026-07-23T03:51:04.700186064Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.700188266Z hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.700190129Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700192031Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700193904Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700195807Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700197556Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.700199508Z hidden_states = self.model(
2026-07-23T03:51:04.700201263Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700203336Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700205463Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700207334Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700209080Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.700211063Z hidden_states, residual = layer(
2026-07-23T03:51:04.700212851Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700214768Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700217261Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700219719Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700221463Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.700223405Z hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.700228613Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700230566Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700232462Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700234414Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700236143Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.700238046Z return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.700239806Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.700241998Z return self.forward_native(x, residual)
2026-07-23T03:51:04.700243823Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.700245757Z return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.700247627Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_static
2026-07-23T03:51:04.700249543Z x = x.float()
2026-07-23T03:51:04.700252832Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.700256610Z
2026-07-23T03:51:04.700258408Z The above exception was the direct cause of the following exception:
2026-07-23T03:51:04.700260214Z
2026-07-23T03:51:04.700261917Z Traceback (most recent call last):
2026-07-23T03:51:04.700263623Z File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
2026-07-23T03:51:04.700265499Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 68, in _log_task_completion
2026-07-23T03:51:04.700268134Z raise AsyncEngineDeadError(
2026-07-23T03:51:04.700269863Z vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.
2026-07-23T03:51:04.702552963Z ERROR: Exception in ASGI application
2026-07-23T03:51:04.702558866Z Traceback (most recent call last):
2026-07-23T03:51:04.702560834Z File "/usr/local/lib/python3.10/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
2026-07-23T03:51:04.702564379Z yield
2026-07-23T03:51:04.702566507Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 271, in __call__
2026-07-23T03:51:04.702571357Z async with anyio.create_task_group() as task_group:
2026-07-23T03:51:04.702573227Z File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
2026-07-23T03:51:04.702575469Z raise BaseExceptionGroup(
2026-07-23T03:51:04.702577231Z exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2026-07-23T03:51:04.702579140Z
2026-07-23T03:51:04.702580984Z During handling of the above exception, another exception occurred:
2026-07-23T03:51:04.702582850Z
2026-07-23T03:51:04.702584544Z Traceback (most recent call last):
2026-07-23T03:51:04.702586535Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-07-23T03:51:04.702588429Z await app(scope, receive, sender)
2026-07-23T03:51:04.702590235Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
2026-07-23T03:51:04.702592118Z await response(scope, receive, send)
2026-07-23T03:51:04.702593877Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 270, in __call__
2026-07-23T03:51:04.702595840Z with collapse_excgroups():
2026-07-23T03:51:04.702597561Z File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
2026-07-23T03:51:04.702599394Z self.gen.throw(typ, value, traceback)
2026-07-23T03:51:04.702601159Z File "/usr/local/lib/python3.10/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
2026-07-23T03:51:04.702603068Z raise exc
2026-07-23T03:51:04.702604874Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 274, in wrap
2026-07-23T03:51:04.702606737Z await func()
2026-07-23T03:51:04.702608502Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 254, in stream_response
2026-07-23T03:51:04.702610428Z async for chunk in self.body_iterator:
2026-07-23T03:51:04.702612210Z File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 846, in chat_completion_stream_generator
2026-07-23T03:51:04.702614205Z await self.engine_client.abort(request_id)
2026-07-23T03:51:04.702615921Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 1243, in abort
2026-07-23T03:51:04.702617838Z raise AsyncEngineDeadError(
2026-07-23T03:51:04.702619831Z vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is not running. If it was running, inspect the output to find the stacktrace of the error that caused the background loop to stop (AsyncEngineDeadError).
2026-07-23T03:51:04.702622281Z
2026-07-23T03:51:04.702623917Z The above exception was the direct cause of the following exception:
2026-07-23T03:51:04.702625718Z
2026-07-23T03:51:04.702627946Z Traceback (most recent call last):
2026-07-23T03:51:04.702643211Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-07-23T03:51:04.702645150Z await app(scope, receive, sender)
2026-07-23T03:51:04.702646977Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 716, in __call__
2026-07-23T03:51:04.702648835Z await self.middleware_stack(scope, receive, send)
2026-07-23T03:51:04.702650643Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 736, in app
2026-07-23T03:51:04.702652476Z await route.handle(scope, receive, send)
2026-07-23T03:51:04.702654253Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 290, in handle
2026-07-23T03:51:04.702656095Z await self.app(scope, receive, send)
2026-07-23T03:51:04.702657830Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 78, in app
2026-07-23T03:51:04.702659880Z await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2026-07-23T03:51:04.702662244Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 56, in wrapped_app
2026-07-23T03:51:04.702665694Z raise RuntimeError("Caught handled exception, but response already started.") from exc
2026-07-23T03:51:04.702667893Z RuntimeError: Caught handled exception, but response already started.
2026-07-23T03:51:04.702669731Z
2026-07-23T03:51:04.702671340Z The above exception was the direct cause of the following exception:
2026-07-23T03:51:04.702673239Z
2026-07-23T03:51:04.702674862Z Traceback (most recent call last):
2026-07-23T03:51:04.702677233Z File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
2026-07-23T03:51:04.702679161Z result = await app( # type: ignore[func-returns-value]
2026-07-23T03:51:04.702680952Z File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2026-07-23T03:51:04.702682811Z return await self.app(scope, receive, send)
2026-07-23T03:51:04.702684557Z File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1082, in __call__
2026-07-23T03:51:04.702686691Z await super().__call__(scope, receive, send)
2026-07-23T03:51:04.702688431Z File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
2026-07-23T03:51:04.702690476Z await self.middleware_stack(scope, receive, send)
2026-07-23T03:51:04.702692252Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
2026-07-23T03:51:04.702694354Z raise exc
2026-07-23T03:51:04.702696382Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
2026-07-23T03:51:04.702698525Z await self.app(scope, receive, _send)
2026-07-23T03:51:04.702700285Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
2026-07-23T03:51:04.702705835Z await self.app(scope, receive, send)
2026-07-23T03:51:04.702708304Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
2026-07-23T03:51:04.702710205Z await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2026-07-23T03:51:04.702712202Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 56, in wrapped_app
2026-07-23T03:51:04.702714127Z raise RuntimeError("Caught handled exception, but response already started.") from exc
2026-07-23T03:51:04.702716055Z RuntimeError: Caught handled exception, but response already started.
2026-07-23T03:51:04.713173377Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.714970623Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.718637314Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.718694821Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.722498257Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.722585244Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.735233268Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:51:04.756590608Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:51:04.768186558Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.768268921Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.781736020Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:51:04.791948299Z INFO: Shutting down
2026-07-23T03:51:04.892200042Z INFO: Waiting for application shutdown.
2026-07-23T03:51:04.892346436Z INFO: Application shutdown complete.
2026-07-23T03:51:04.893243235Z INFO: Finished server process [1]
2026-07-23T03:51:06.253648359Z INFO 07-23 03:51:06 multiproc_worker_utils.py:121] Killing local vLLM worker processes
2026-07-23T03:51:10.732959761Z Future exception was never retrieved
2026-07-23T03:51:10.732970009Z future: <Future finished exception=OutOfMemoryError('CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')>
2026-07-23T03:51:10.732975335Z Traceback (most recent call last):
2026-07-23T03:51:10.732983365Z File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 466, in chat_completion_stream_generator
2026-07-23T03:51:10.732985745Z async for res in result_generator:
2026-07-23T03:51:10.732987633Z File "/usr/local/corex/lib/python3/dist-packages/vllm/utils.py", line 458, in iterate_with_cancellation
2026-07-23T03:51:10.732989646Z item = await awaits[0]
2026-07-23T03:51:10.732991495Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 1046, in generate
2026-07-23T03:51:10.732993455Z async for output in await self.add_request(
2026-07-23T03:51:10.732995398Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 114, in generator
2026-07-23T03:51:10.732997331Z raise result
2026-07-23T03:51:10.732999159Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 56, in _log_task_completion
2026-07-23T03:51:10.733001089Z return_value = task.result()
2026-07-23T03:51:10.733002909Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 857, in run_engine_loop
2026-07-23T03:51:10.733004845Z result = task.result()
2026-07-23T03:51:10.733006732Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 776, in engine_step
2026-07-23T03:51:10.733008659Z request_outputs = await self.engine.step_async(virtual_engine)
2026-07-23T03:51:10.733010626Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 348, in step_async
2026-07-23T03:51:10.733012560Z outputs = await self.model_executor.execute_model_async(
2026-07-23T03:51:10.733014593Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 181, in execute_model_async
2026-07-23T03:51:10.733016583Z return await self._driver_execute_model_async(execute_model_req)
2026-07-23T03:51:10.733018783Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 224, in _driver_execute_model_async
2026-07-23T03:51:10.733020797Z return await self.driver_exec_model(execute_model_req)
2026-07-23T03:51:10.733022570Z File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2026-07-23T03:51:10.733024502Z result = self.fn(*self.args, **self.kwargs)
2026-07-23T03:51:10.733026262Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:10.733028655Z output = self.model_runner.execute_model(
2026-07-23T03:51:10.733030678Z File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:10.733032715Z return func(*args, **kwargs)
2026-07-23T03:51:10.733034659Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:10.733039337Z hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:10.733041153Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733043068Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733044905Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733046777Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733050243Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:10.733053412Z hidden_states = self.model(
2026-07-23T03:51:10.733057551Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733059640Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733061352Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733063252Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733065023Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:10.733067165Z hidden_states, residual = layer(
2026-07-23T03:51:10.733069092Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733071000Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733072984Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733074867Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733076607Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:10.733078748Z hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:10.733080781Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733082662Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733084425Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733086295Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733088148Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:10.733090040Z return self._forward_method(*args, **kwargs)
2026-07-23T03:51:10.733091807Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:10.733093814Z return self.forward_native(x, residual)
2026-07-23T03:51:10.733095539Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:10.733101307Z return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:10.733104565Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_static
2026-07-23T03:51:10.733107808Z x = x.float()
2026-07-23T03:51:10.733109873Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:10.735532351Z Task exception was never retrieved
2026-07-23T03:51:10.735545848Z future: <Task finished name='Task-129921' coro=<MultiprocessingGPUExecutorAsync._start_worker_execution_loop() done, defined at /usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py:253> exception=OutOfMemoryError('CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')>
2026-07-23T03:51:10.735550322Z Traceback (most recent call last):
2026-07-23T03:51:10.735552351Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 258, in _start_worker_execution_loop
2026-07-23T03:51:10.735554626Z return await asyncio.gather(*coros)
2026-07-23T03:51:10.735556397Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 183, in execute_method_async
2026-07-23T03:51:10.735558462Z return await future
2026-07-23T03:51:10.735560338Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T02:29:15.072467456Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:15 model_runner.py:1074] Starting to load model /model...
2026-07-23T02:29:15.123205314Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.123647472Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.136250183Z INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.142620520Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:15 selector.py:115] Using XFormers backend.
2026-07-23T02:29:15.267061728Z
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
2026-07-23T02:29:16.504035279Z
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:01<00:18, 1.24s/it]
2026-07-23T02:29:17.560515507Z
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:02<00:15, 1.13s/it]
2026-07-23T02:29:18.454967452Z
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:03<00:13, 1.02s/it]
2026-07-23T02:29:19.622930900Z
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:04<00:12, 1.08s/it]
2026-07-23T02:29:20.607005016Z
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:05<00:11, 1.05s/it]
2026-07-23T02:29:21.663620433Z
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:06<00:10, 1.05s/it]
2026-07-23T02:29:22.762559585Z
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:07<00:09, 1.07s/it]
2026-07-23T02:29:23.920513395Z
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:08<00:08, 1.09s/it]
2026-07-23T02:29:25.327235482Z
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:10<00:08, 1.19s/it]
2026-07-23T02:29:26.537079790Z
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:11<00:07, 1.20s/it]
2026-07-23T02:29:27.538590658Z
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:12<00:05, 1.14s/it]
2026-07-23T02:29:28.909915510Z
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:13<00:04, 1.21s/it]
2026-07-23T02:29:30.177921768Z
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:14<00:03, 1.23s/it]
2026-07-23T02:29:31.144529015Z
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:15<00:02, 1.15s/it]
2026-07-23T02:29:32.170218873Z
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:16<00:01, 1.11s/it]
2026-07-23T02:29:33.339901076Z
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:18<00:00, 1.13s/it]
2026-07-23T02:29:33.340059995Z
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:18<00:00, 1.13s/it]
2026-07-23T02:29:33.340067531Z
2026-07-23T02:29:33.791755457Z (VllmWorkerProcess pid=345) INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:29:33.791764238Z (VllmWorkerProcess pid=344) INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:29:33.792212417Z (VllmWorkerProcess pid=346) INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:29:33.799049642Z INFO 07-23 02:29:33 model_runner.py:1085] Loading model weights took 17.3529 GB
2026-07-23T02:32:25.902136731Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.903895709Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.905723523Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.907479743Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.915921136Z INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.918179567Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:25 corex_gdn.py:56] Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so
2026-07-23T02:32:25.919624182Z INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.921789252Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:25 corex_gdn.py:228] Using fused CoreX GDN prefill operator
2026-07-23T02:32:25.950838770Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:25.950882505Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:25.950908225Z INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:25.950931658Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:25 corex_moe.py:339] Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma
2026-07-23T02:32:26.045895062Z (VllmWorkerProcess pid=346) INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:26.045990261Z (VllmWorkerProcess pid=345) INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:26.046000888Z (VllmWorkerProcess pid=344) INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:26.046049182Z INFO 07-23 02:32:26 corex_fa2.py:333] Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 max_q=2048 max_k=2048
2026-07-23T02:32:27.731056211Z INFO 07-23 02:32:27 distributed_gpu_executor.py:57] # GPU blocks: 19259, # CPU blocks: 6553
2026-07-23T02:32:27.731106894Z INFO 07-23 02:32:27 distributed_gpu_executor.py:61] Maximum concurrency for 256000 tokens per request: 1.20x
2026-07-23T02:32:31.498383100Z INFO 07-23 02:32:31 serving_chat.py:107] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
2026-07-23T02:32:31.499707957Z INFO 07-23 02:32:31 serving_chat.py:129] Reasoning parser 'qwen3' enabled.
2026-07-23T02:32:31.499774658Z WARNING 07-23 02:32:31 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
2026-07-23T02:32:31.499825780Z INFO 07-23 02:32:31 launcher.py:19] Available routes are:
2026-07-23T02:32:31.499867858Z INFO 07-23 02:32:31 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
2026-07-23T02:32:31.499886381Z INFO 07-23 02:32:31 launcher.py:27] Route: /docs, Methods: HEAD, GET
2026-07-23T02:32:31.499904662Z INFO 07-23 02:32:31 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
2026-07-23T02:32:31.499945605Z INFO 07-23 02:32:31 launcher.py:27] Route: /redoc, Methods: HEAD, GET
2026-07-23T02:32:31.499951625Z INFO 07-23 02:32:31 launcher.py:27] Route: /health, Methods: GET
2026-07-23T02:32:31.499989247Z INFO 07-23 02:32:31 launcher.py:27] Route: /tokenize, Methods: POST
2026-07-23T02:32:31.499991563Z INFO 07-23 02:32:31 launcher.py:27] Route: /detokenize, Methods: POST
2026-07-23T02:32:31.500017919Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/models, Methods: GET
2026-07-23T02:32:31.500027643Z INFO 07-23 02:32:31 launcher.py:27] Route: /version, Methods: GET
2026-07-23T02:32:31.500047221Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/chat/completions, Methods: POST
2026-07-23T02:32:31.500071390Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/completions, Methods: POST
2026-07-23T02:32:31.500084135Z INFO 07-23 02:32:31 launcher.py:27] Route: /v1/embeddings, Methods: POST
2026-07-23T02:32:31.517724005Z INFO: Started server process [1]
2026-07-23T02:32:31.517775355Z INFO: Waiting for application startup.
2026-07-23T02:32:31.518104234Z INFO: Application startup complete.
2026-07-23T02:32:31.518434320Z INFO: Uvicorn running on socket ('0.0.0.0', 8000) (Press CTRL+C to quit)
2026-07-23T02:32:41.518080032Z INFO 07-23 02:32:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:32:44.976380698Z INFO: 172.17.0.1:34848 - "GET /health HTTP/1.1" 200 OK
2026-07-23T02:32:44.977131890Z INFO: 172.17.0.1:34848 - "GET /v1/models HTTP/1.1" 200 OK
2026-07-23T02:32:51.517794437Z INFO 07-23 02:32:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:01.518176592Z INFO 07-23 02:33:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:11.517753255Z INFO 07-23 02:33:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:21.518188707Z INFO 07-23 02:33:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:31.518515223Z INFO 07-23 02:33:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:41.518939596Z INFO 07-23 02:33:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:33:51.519425845Z INFO 07-23 02:33:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:01.519896974Z INFO 07-23 02:34:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:11.520418085Z INFO 07-23 02:34:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:21.520940369Z INFO 07-23 02:34:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:31.521337663Z INFO 07-23 02:34:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:41.521711416Z INFO 07-23 02:34:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:34:51.522131821Z INFO 07-23 02:34:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:01.522501314Z INFO 07-23 02:35:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:11.522857298Z INFO 07-23 02:35:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:21.523285280Z INFO 07-23 02:35:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:31.522702429Z INFO 07-23 02:35:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:41.523044729Z INFO 07-23 02:35:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:35:51.523462651Z INFO 07-23 02:35:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:01.523885993Z INFO 07-23 02:36:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:11.524260127Z INFO 07-23 02:36:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:21.523723862Z INFO 07-23 02:36:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:31.524139239Z INFO 07-23 02:36:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:41.523592375Z INFO 07-23 02:36:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:36:51.523995257Z INFO 07-23 02:36:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:01.524428577Z INFO 07-23 02:37:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:11.525008725Z INFO 07-23 02:37:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:21.524752003Z INFO 07-23 02:37:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:31.525171202Z INFO 07-23 02:37:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:35.774341598Z INFO: 172.28.22.23:22372 - "GET /get_model_info HTTP/1.1" 404 Not Found
2026-07-23T02:37:35.776361569Z INFO: 172.28.22.23:43360 - "GET /get_server_info HTTP/1.1" 404 Not Found
2026-07-23T02:37:35.777756478Z INFO: 172.28.22.23:30571 - "GET /version HTTP/1.1" 200 OK
2026-07-23T02:37:36.257909619Z /usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2026-07-23T02:37:36.257934560Z from pkg_resources import resource_string
2026-07-23T02:37:37.889916076Z (VllmWorkerProcess pid=346) INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.889927964Z INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.889989104Z (VllmWorkerProcess pid=344) INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.889993203Z (VllmWorkerProcess pid=345) INFO 07-23 02:37:37 corex_fa2.py:507] Using CoreX paged FA2 chunked prefill: B=1 Hq=4 Hkv=1 D=256 max_q=17 cache_blocks=2
2026-07-23T02:37:37.970977734Z INFO 07-23 02:37:37 metrics.py:345] Avg prompt throughput: 2.6 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:37.977175350Z (VllmWorkerProcess pid=346) INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977199535Z (VllmWorkerProcess pid=344) INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977216621Z INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977220191Z (VllmWorkerProcess pid=345) INFO 07-23 02:37:37 corex_gdn.py:138] Using fused CoreX GDN decode operator
2026-07-23T02:37:37.977862426Z (VllmWorkerProcess pid=346) INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:37.977866130Z (VllmWorkerProcess pid=345) INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:37.977868711Z INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:37.977871546Z (VllmWorkerProcess pid=344) INFO 07-23 02:37:37 corex_moe.py:249] Using CoreX fused MoE decode operator
2026-07-23T02:37:43.000160753Z INFO 07-23 02:37:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:44.269161765Z INFO: 172.28.22.23:13116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:37:44.396724776Z INFO: 172.28.22.23:7057 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:37:48.010705708Z INFO 07-23 02:37:48 metrics.py:345] Avg prompt throughput: 57.9 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:49.136976286Z INFO: 172.28.22.23:45259 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:37:53.025290553Z INFO 07-23 02:37:53 metrics.py:345] Avg prompt throughput: 7.2 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:37:58.061514748Z INFO 07-23 02:37:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:03.073788209Z INFO 07-23 02:38:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:06.918087930Z INFO: 172.28.22.23:34227 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:08.377827090Z INFO 07-23 02:38:08 metrics.py:345] Avg prompt throughput: 4.3 tokens/s, Avg generation throughput: 15.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:13.046591181Z INFO: 172.28.22.23:29692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:13.172739732Z INFO: 172.28.22.23:48540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:13.710792259Z INFO 07-23 02:38:13 metrics.py:345] Avg prompt throughput: 58.3 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:14.666143261Z INFO: 172.28.22.23:34379 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:38:18.720975352Z INFO 07-23 02:38:18 metrics.py:345] Avg prompt throughput: 74.2 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:23.725539749Z INFO 07-23 02:38:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:28.760348109Z INFO 07-23 02:38:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:33.763160416Z INFO 07-23 02:38:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:38.786603579Z INFO 07-23 02:38:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:43.822945213Z INFO 07-23 02:38:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:48.829240975Z INFO 07-23 02:38:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:53.861865865Z INFO 07-23 02:38:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:38:58.900819345Z INFO 07-23 02:38:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:03.930669117Z INFO 07-23 02:39:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:08.932255487Z INFO 07-23 02:39:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:13.947463156Z INFO 07-23 02:39:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:17.129519291Z INFO: 172.28.22.23:47285 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:21.526611923Z INFO 07-23 02:39:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:22.789790337Z ERROR 07-23 02:39:22 serving_chat.py:227] Error in loading multi-modal data
2026-07-23T02:39:22.789821427Z ERROR 07-23 02:39:22 serving_chat.py:227] Traceback (most recent call last):
2026-07-23T02:39:22.789824505Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 770, in _request
2026-07-23T02:39:22.789827021Z ERROR 07-23 02:39:22 serving_chat.py:227] resp = await handler(req)
2026-07-23T02:39:22.789829231Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
2026-07-23T02:39:22.789831554Z ERROR 07-23 02:39:22 serving_chat.py:227] conn = await self._connector.connect(
2026-07-23T02:39:22.789833480Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 642, in connect
2026-07-23T02:39:22.789835993Z ERROR 07-23 02:39:22 serving_chat.py:227] proto = await self._create_connection(req, traces, timeout)
2026-07-23T02:39:22.789838507Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1209, in _create_connection
2026-07-23T02:39:22.789840743Z ERROR 07-23 02:39:22 serving_chat.py:227] _, proto = await self._create_direct_connection(req, traces, timeout)
2026-07-23T02:39:22.789842704Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1550, in _create_direct_connection
2026-07-23T02:39:22.789844833Z ERROR 07-23 02:39:22 serving_chat.py:227] transp, proto = await self._wrap_create_connection(
2026-07-23T02:39:22.789857869Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1283, in _wrap_create_connection
2026-07-23T02:39:22.789859986Z ERROR 07-23 02:39:22 serving_chat.py:227] return await self._loop.create_connection(*args, **kwargs, sock=sock)
2026-07-23T02:39:22.789861916Z ERROR 07-23 02:39:22 serving_chat.py:227] File "uvloop/loop.pyx", line 2088, in create_connection
2026-07-23T02:39:22.789864184Z ERROR 07-23 02:39:22 serving_chat.py:227] File "uvloop/loop.pyx", line 2083, in uvloop.loop.Loop.create_connection
2026-07-23T02:39:22.789866163Z ERROR 07-23 02:39:22 serving_chat.py:227] asyncio.exceptions.CancelledError
2026-07-23T02:39:22.789867986Z ERROR 07-23 02:39:22 serving_chat.py:227]
2026-07-23T02:39:22.789878123Z ERROR 07-23 02:39:22 serving_chat.py:227] The above exception was the direct cause of the following exception:
2026-07-23T02:39:22.789887569Z ERROR 07-23 02:39:22 serving_chat.py:227]
2026-07-23T02:39:22.789890593Z ERROR 07-23 02:39:22 serving_chat.py:227] Traceback (most recent call last):
2026-07-23T02:39:22.789892686Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 225, in create_chat_completion
2026-07-23T02:39:22.789894923Z ERROR 07-23 02:39:22 serving_chat.py:227] mm_data = await mm_data_future
2026-07-23T02:39:22.789896983Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 248, in all_mm_data
2026-07-23T02:39:22.789899260Z ERROR 07-23 02:39:22 serving_chat.py:227] items = await asyncio.gather(*self._items)
2026-07-23T02:39:22.789901151Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/multimodal/utils.py", line 140, in async_get_and_parse_image
2026-07-23T02:39:22.789903256Z ERROR 07-23 02:39:22 serving_chat.py:227] image = await async_fetch_image(image_url)
2026-07-23T02:39:22.789905277Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/multimodal/utils.py", line 62, in async_fetch_image
2026-07-23T02:39:22.789907335Z ERROR 07-23 02:39:22 serving_chat.py:227] image_raw = await global_http_connection.async_get_bytes(
2026-07-23T02:39:22.789909226Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/corex/lib/python3/dist-packages/vllm/connections.py", line 92, in async_get_bytes
2026-07-23T02:39:22.789911211Z ERROR 07-23 02:39:22 serving_chat.py:227] async with await self.get_async_response(url, timeout=timeout) as r:
2026-07-23T02:39:22.789913906Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1488, in __aenter__
2026-07-23T02:39:22.789915849Z ERROR 07-23 02:39:22 serving_chat.py:227] self._resp: _RetType = await self._coro
2026-07-23T02:39:22.789918323Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 622, in _request
2026-07-23T02:39:22.789921398Z ERROR 07-23 02:39:22 serving_chat.py:227] with timer:
2026-07-23T02:39:22.789929127Z ERROR 07-23 02:39:22 serving_chat.py:227] File "/usr/local/lib/python3.10/site-packages/aiohttp/helpers.py", line 685, in __exit__
2026-07-23T02:39:22.789931189Z ERROR 07-23 02:39:22 serving_chat.py:227] raise asyncio.TimeoutError from exc_val
2026-07-23T02:39:22.789933347Z ERROR 07-23 02:39:22 serving_chat.py:227] asyncio.exceptions.TimeoutError
2026-07-23T02:39:22.789982206Z INFO: 172.28.22.23:63905 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:39:23.339790176Z INFO: 172.28.22.23:48915 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:23.862218623Z INFO: 172.28.22.23:39188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:26.537243563Z INFO 07-23 02:39:26 metrics.py:345] Avg prompt throughput: 12.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:31.709534191Z INFO: 172.28.22.23:30519 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:31.709975900Z INFO 07-23 02:39:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:32.540087790Z INFO: 172.28.22.23:28182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:36.711389406Z INFO 07-23 02:39:36 metrics.py:345] Avg prompt throughput: 6.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:38.834910565Z INFO: 172.28.22.23:25497 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:40.106985905Z INFO: 172.28.22.23:31187 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:41.375383434Z INFO: 172.28.22.23:1054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:41.736972408Z INFO 07-23 02:39:41 metrics.py:345] Avg prompt throughput: 6.6 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:42.644170272Z INFO: 172.28.22.23:10283 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:43.909888974Z INFO: 172.28.22.23:43195 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:44.026597768Z INFO: 172.28.22.23:31826 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:39:45.279682681Z INFO: 172.28.22.23:31890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:46.532238457Z INFO: 172.28.22.23:52470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:46.768052149Z INFO 07-23 02:39:46 metrics.py:345] Avg prompt throughput: 8.7 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:47.779232275Z INFO: 172.28.22.23:1935 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:47.894850660Z INFO: 172.28.22.23:24799 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:39:49.140115276Z INFO: 172.28.22.23:32610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:50.385833050Z INFO: 172.28.22.23:34946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:51.632711881Z INFO: 172.28.22.23:29640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:51.824060918Z INFO 07-23 02:39:51 metrics.py:345] Avg prompt throughput: 8.7 tokens/s, Avg generation throughput: 15.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:52.883677883Z INFO: 172.28.22.23:25348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:54.134965370Z INFO: 172.28.22.23:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:55.381846551Z INFO: 172.28.22.23:9523 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:56.629565470Z INFO: 172.28.22.23:56394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:39:56.827523945Z INFO 07-23 02:39:56 metrics.py:345] Avg prompt throughput: 8.8 tokens/s, Avg generation throughput: 16.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:39:58.129748260Z INFO: 172.28.22.23:9093 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:01.848026951Z INFO 07-23 02:40:01 metrics.py:345] Avg prompt throughput: 2.8 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:06.858910882Z INFO 07-23 02:40:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:11.862489179Z INFO 07-23 02:40:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:16.899163580Z INFO 07-23 02:40:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:21.910058417Z INFO 07-23 02:40:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:26.929209213Z INFO 07-23 02:40:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:31.962337057Z INFO 07-23 02:40:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:36.964040774Z INFO 07-23 02:40:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:41.993035873Z INFO 07-23 02:40:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:43.624005510Z INFO: 172.28.22.23:2587 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:44.061237847Z INFO: 172.28.22.23:6084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:47.185504184Z INFO: 172.28.22.23:50569 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:40:47.377278073Z INFO 07-23 02:40:47 metrics.py:345] Avg prompt throughput: 7.8 tokens/s, Avg generation throughput: 17.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:52.412861124Z INFO 07-23 02:40:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:40:57.415707222Z INFO 07-23 02:40:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:02.437584264Z INFO 07-23 02:41:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:07.476637249Z INFO 07-23 02:41:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:12.517176365Z INFO 07-23 02:41:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:17.531413276Z INFO 07-23 02:41:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:22.564441104Z INFO 07-23 02:41:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:27.577151702Z INFO 07-23 02:41:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:32.611584807Z INFO 07-23 02:41:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:37.623284038Z INFO 07-23 02:41:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:42.639031894Z INFO 07-23 02:41:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:47.641862259Z INFO 07-23 02:41:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:52.683325904Z INFO 07-23 02:41:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:41:57.720443718Z INFO 07-23 02:41:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:02.748390108Z INFO 07-23 02:42:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:07.784213915Z INFO 07-23 02:42:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:12.827245969Z INFO 07-23 02:42:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:17.859544046Z INFO 07-23 02:42:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:22.889429150Z INFO 07-23 02:42:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:27.923212049Z INFO 07-23 02:42:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:32.925156292Z INFO 07-23 02:42:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:37.955994920Z INFO 07-23 02:42:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:40.293610439Z INFO: 172.28.22.23:17026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:40.411301190Z INFO: 172.28.22.23:47159 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:42:40.524865477Z INFO: 172.28.22.23:53783 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:42:40.637743018Z INFO: 172.28.22.23:24159 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:42:41.181611635Z INFO: 172.28.22.23:38467 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:41.727889239Z INFO: 172.28.22.23:24963 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:42.962296089Z INFO 07-23 02:42:42 metrics.py:345] Avg prompt throughput: 64.9 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:42:43.783197309Z INFO: 172.28.22.23:14284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:44.459669799Z INFO: 172.28.22.23:49964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:44.575803355Z INFO: 172.28.22.23:54360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:42:47.973780549Z INFO 07-23 02:42:47 metrics.py:345] Avg prompt throughput: 15.0 tokens/s, Avg generation throughput: 18.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:01.858799064Z INFO 07-23 02:43:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:10.553432764Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:04<00:09, 4.87s/it]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:04<00:00, 1.31s/it]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:05<00:00, 1.69s/it]
2026-07-23T02:43:10.569991401Z INFO 07-23 02:43:10 metrics.py:345] Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:11.990491527Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.23it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.17it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 6.98it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.52it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.10it/s]
2026-07-23T02:43:13.445065774Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.22it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.22it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.07it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.63it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.14it/s]
2026-07-23T02:43:14.876448068Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.23it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.28it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.15it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.73it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.23it/s]
2026-07-23T02:43:15.701717222Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.19it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.19it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.69it/s]
2026-07-23T02:43:15.720137025Z INFO 07-23 02:43:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:16.473274583Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.21it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 5.71it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.28it/s]
2026-07-23T02:43:18.904638207Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.14it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.13it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 6.94it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.55it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 11.88it/s]
Compiling FSM index for all state transitions: 48%|████▊ | 11/23 [00:01<00:00, 13.84it/s]
Compiling FSM index for all state transitions: 57%|█████▋ | 13/23 [00:01<00:00, 15.41it/s]
Compiling FSM index for all state transitions: 65%|██████▌ | 15/23 [00:01<00:00, 16.62it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.54it/s]
Compiling FSM index for all state transitions: 87%|████████▋ | 20/23 [00:01<00:00, 18.47it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:01<00:00, 19.06it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.89it/s]
2026-07-23T02:43:21.350197196Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.14it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.24it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 7.05it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.67it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 11.99it/s]
Compiling FSM index for all state transitions: 52%|█████▏ | 12/23 [00:01<00:00, 14.60it/s]
Compiling FSM index for all state transitions: 61%|██████ | 14/23 [00:01<00:00, 15.87it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.26it/s]
Compiling FSM index for all state transitions: 87%|████████▋ | 20/23 [00:01<00:00, 18.16it/s]
Compiling FSM index for all state transitions: 96%|█████████▌| 22/23 [00:01<00:00, 18.40it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.89it/s]
2026-07-23T02:43:21.397139672Z INFO 07-23 02:43:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:22.261063391Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.22it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.24it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.73it/s]
2026-07-23T02:43:22.957955678Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.31it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.77it/s]
2026-07-23T02:43:24.351851677Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.25it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.32it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.21it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.80it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.26it/s]
2026-07-23T02:43:25.800354211Z
Compiling FSM index for all state transitions: 0%| | 0/7 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 1/7 [00:00<00:02, 2.16it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 3/7 [00:00<00:00, 4.28it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 5/7 [00:00<00:00, 7.15it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:00<00:00, 9.73it/s]
Compiling FSM index for all state transitions: 100%|██████████| 7/7 [00:01<00:00, 5.16it/s]
2026-07-23T02:43:26.709113498Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.23it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.73it/s]
2026-07-23T02:43:26.727308641Z INFO 07-23 02:43:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:27.404963736Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.30it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.78it/s]
2026-07-23T02:43:29.815689817Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.14it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.21it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 7.04it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.64it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 11.95it/s]
Compiling FSM index for all state transitions: 48%|████▊ | 11/23 [00:01<00:00, 13.88it/s]
Compiling FSM index for all state transitions: 57%|█████▋ | 13/23 [00:01<00:00, 15.42it/s]
Compiling FSM index for all state transitions: 65%|██████▌ | 15/23 [00:01<00:00, 16.60it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.53it/s]
Compiling FSM index for all state transitions: 87%|████████▋ | 20/23 [00:01<00:00, 18.46it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:01<00:00, 18.81it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.91it/s]
2026-07-23T02:43:31.881017891Z INFO 07-23 02:43:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:32.258419554Z
Compiling FSM index for all state transitions: 0%| | 0/23 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 4%|▍ | 1/23 [00:00<00:10, 2.13it/s]
Compiling FSM index for all state transitions: 13%|█▎ | 3/23 [00:00<00:04, 4.25it/s]
Compiling FSM index for all state transitions: 22%|██▏ | 5/23 [00:00<00:02, 7.10it/s]
Compiling FSM index for all state transitions: 30%|███ | 7/23 [00:00<00:01, 9.72it/s]
Compiling FSM index for all state transitions: 39%|███▉ | 9/23 [00:01<00:01, 12.03it/s]
Compiling FSM index for all state transitions: 48%|████▊ | 11/23 [00:01<00:00, 13.97it/s]
Compiling FSM index for all state transitions: 57%|█████▋ | 13/23 [00:01<00:00, 15.50it/s]
Compiling FSM index for all state transitions: 65%|██████▌ | 15/23 [00:01<00:00, 16.69it/s]
Compiling FSM index for all state transitions: 74%|███████▍ | 17/23 [00:01<00:00, 17.58it/s]
Compiling FSM index for all state transitions: 83%|████████▎ | 19/23 [00:01<00:00, 18.25it/s]
Compiling FSM index for all state transitions: 96%|█████████▌| 22/23 [00:01<00:00, 18.93it/s]
Compiling FSM index for all state transitions: 100%|██████████| 23/23 [00:02<00:00, 9.89it/s]
2026-07-23T02:43:32.983539758Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.24it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.29it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.76it/s]
2026-07-23T02:43:33.672113037Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.32it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.78it/s]
2026-07-23T02:43:34.368060101Z
Compiling FSM index for all state transitions: 0%| | 0/3 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 33%|███▎ | 1/3 [00:00<00:00, 2.24it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 6.28it/s]
Compiling FSM index for all state transitions: 100%|██████████| 3/3 [00:00<00:00, 4.76it/s]
2026-07-23T02:43:34.941584523Z
Compiling FSM index for all state transitions: 0%| | 0/2 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 50%|█████ | 1/2 [00:00<00:00, 2.25it/s]
Compiling FSM index for all state transitions: 50%|█████ | 1/2 [00:00<00:00, 1.96it/s]
2026-07-23T02:43:35.230436592Z INFO: 172.28.22.23:18839 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:38.310835204Z
Compiling FSM index for all state transitions: 0%| | 0/35 [00:00<?, ?it/s]
Compiling FSM index for all state transitions: 3%|▎ | 1/35 [00:00<00:16, 2.11it/s]
Compiling FSM index for all state transitions: 9%|▊ | 3/35 [00:00<00:05, 6.13it/s]
Compiling FSM index for all state transitions: 14%|█▍ | 5/35 [00:00<00:03, 9.43it/s]
Compiling FSM index for all state transitions: 20%|██ | 7/35 [00:00<00:02, 12.06it/s]
Compiling FSM index for all state transitions: 26%|██▌ | 9/35 [00:00<00:01, 13.89it/s]
Compiling FSM index for all state transitions: 31%|███▏ | 11/35 [00:01<00:01, 15.18it/s]
Compiling FSM index for all state transitions: 37%|███▋ | 13/35 [00:01<00:02, 10.52it/s]
Compiling FSM index for all state transitions: 43%|████▎ | 15/35 [00:01<00:01, 12.25it/s]
Compiling FSM index for all state transitions: 49%|████▊ | 17/35 [00:01<00:01, 13.80it/s]
Compiling FSM index for all state transitions: 54%|█████▍ | 19/35 [00:01<00:01, 14.75it/s]
Compiling FSM index for all state transitions: 60%|██████ | 21/35 [00:01<00:00, 15.66it/s]
Compiling FSM index for all state transitions: 66%|██████▌ | 23/35 [00:01<00:00, 16.67it/s]
Compiling FSM index for all state transitions: 71%|███████▏ | 25/35 [00:01<00:00, 17.44it/s]
Compiling FSM index for all state transitions: 77%|███████▋ | 27/35 [00:02<00:00, 17.73it/s]
Compiling FSM index for all state transitions: 83%|████████▎ | 29/35 [00:02<00:00, 17.93it/s]
Compiling FSM index for all state transitions: 89%|████████▊ | 31/35 [00:02<00:00, 18.39it/s]
Compiling FSM index for all state transitions: 94%|█████████▍| 33/35 [00:02<00:00, 18.07it/s]
Compiling FSM index for all state transitions: 100%|██████████| 35/35 [00:02<00:00, 18.50it/s]
Compiling FSM index for all state transitions: 100%|██████████| 35/35 [00:02<00:00, 11.94it/s]
2026-07-23T02:43:38.462680310Z INFO 07-23 02:43:38 metrics.py:345] Avg prompt throughput: 3.5 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:39.428082640Z INFO: 172.28.22.23:41311 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:41.354629107Z INFO: 172.28.22.23:35141 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:41.951932899Z INFO: 172.28.22.23:40937 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:42.708616569Z INFO: 172.28.22.23:3677 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:43.378056127Z INFO: 172.28.22.23:46863 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:43.593932801Z INFO 07-23 02:43:43 metrics.py:345] Avg prompt throughput: 22.6 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:43:44.482046916Z INFO: 172.28.22.23:46268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:43:51.952117180Z INFO 07-23 02:43:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:01.952434751Z INFO 07-23 02:44:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:11.953116612Z INFO 07-23 02:44:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:21.953465417Z INFO 07-23 02:44:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:31.954404947Z INFO 07-23 02:44:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:41.955602923Z INFO 07-23 02:44:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:44:51.956600553Z INFO 07-23 02:44:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:01.429424131Z INFO 07-23 02:45:01 metrics.py:345] Avg prompt throughput: 33.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:02.020271472Z INFO: 172.28.22.23:10993 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:02.023173066Z INFO: 172.28.22.23:34691 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:02.542033396Z INFO: 172.28.22.23:44245 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:03.031598393Z INFO: 172.28.22.23:5495 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:03.034079308Z INFO: 172.28.22.23:33361 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:03.035816648Z INFO: 172.28.22.23:1279 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:03.038855639Z ERROR 07-23 02:45:03 serving_chat.py:221] Error in applying chat template from request
2026-07-23T02:45:03.038865861Z ERROR 07-23 02:45:03 serving_chat.py:221] Traceback (most recent call last):
2026-07-23T02:45:03.038868246Z ERROR 07-23 02:45:03 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 210, in create_chat_completion
2026-07-23T02:45:03.038870615Z ERROR 07-23 02:45:03 serving_chat.py:221] prompt = apply_hf_chat_template(
2026-07-23T02:45:03.038872879Z ERROR 07-23 02:45:03 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 574, in apply_hf_chat_template
2026-07-23T02:45:03.038875232Z ERROR 07-23 02:45:03 serving_chat.py:221] return tokenizer.apply_chat_template(
2026-07-23T02:45:03.038877096Z ERROR 07-23 02:45:03 serving_chat.py:221] File "/usr/local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1623, in apply_chat_template
2026-07-23T02:45:03.038879145Z ERROR 07-23 02:45:03 serving_chat.py:221] isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
2026-07-23T02:45:03.038881206Z ERROR 07-23 02:45:03 serving_chat.py:221] IndexError: list index out of range
2026-07-23T02:45:03.039113159Z /usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py:316: RuntimeWarning: coroutine 'AsyncMultiModalItemTracker.all_mm_data' was never awaited
2026-07-23T02:45:03.039116978Z generator = await chat(raw_request).create_chat_completion(
2026-07-23T02:45:03.039119691Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback
2026-07-23T02:45:03.039249660Z INFO: 172.28.22.23:39148 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:45:12.020995535Z INFO 07-23 02:45:12 metrics.py:345] Avg prompt throughput: 3.6 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:18.489898901Z INFO: 172.28.22.23:47283 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:45:18.707020154Z INFO 07-23 02:45:18 metrics.py:345] Avg prompt throughput: 5.8 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:23.747292153Z INFO 07-23 02:45:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:28.783886861Z INFO 07-23 02:45:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:33.805881768Z INFO 07-23 02:45:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:38.828813167Z INFO 07-23 02:45:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:43.837844599Z INFO 07-23 02:45:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:48.863019622Z INFO 07-23 02:45:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:53.902955303Z INFO 07-23 02:45:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:45:58.919192631Z INFO 07-23 02:45:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:03.934701110Z INFO 07-23 02:46:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:08.965144483Z INFO 07-23 02:46:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:13.969498143Z INFO 07-23 02:46:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:19.000136074Z INFO 07-23 02:46:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:24.035080024Z INFO 07-23 02:46:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:29.054704172Z INFO 07-23 02:46:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:34.075751226Z INFO 07-23 02:46:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:39.109012197Z INFO 07-23 02:46:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:44.142536722Z INFO 07-23 02:46:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:49.151030740Z INFO 07-23 02:46:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:54.155952934Z INFO 07-23 02:46:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:46:59.168320199Z INFO 07-23 02:46:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:04.198850701Z INFO 07-23 02:47:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:09.227769526Z INFO 07-23 02:47:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:14.236849600Z INFO 07-23 02:47:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:19.250060510Z INFO 07-23 02:47:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:24.278224796Z INFO 07-23 02:47:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:29.302744374Z INFO 07-23 02:47:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:34.340169258Z INFO 07-23 02:47:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:39.373263942Z INFO 07-23 02:47:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:44.400614703Z INFO 07-23 02:47:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:49.429303585Z INFO 07-23 02:47:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:54.454631971Z INFO 07-23 02:47:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:47:59.476270315Z INFO 07-23 02:47:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:04.489781001Z INFO 07-23 02:48:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:09.502845316Z INFO 07-23 02:48:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:14.525253044Z INFO 07-23 02:48:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:19.535460002Z INFO 07-23 02:48:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:24.549336142Z INFO 07-23 02:48:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:29.562276558Z INFO 07-23 02:48:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:34.582693092Z INFO 07-23 02:48:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:39.606402177Z INFO 07-23 02:48:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:44.618006935Z INFO 07-23 02:48:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:49.639887326Z INFO 07-23 02:48:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:54.661553164Z INFO 07-23 02:48:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:48:59.691872658Z INFO 07-23 02:48:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:04.716442787Z INFO 07-23 02:49:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:09.737600900Z INFO 07-23 02:49:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:14.765495628Z INFO 07-23 02:49:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:19.785966586Z INFO 07-23 02:49:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:24.808905236Z INFO 07-23 02:49:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:29.823237965Z INFO 07-23 02:49:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:34.840213253Z INFO 07-23 02:49:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:39.860991050Z INFO 07-23 02:49:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:44.886401826Z INFO 07-23 02:49:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:49.902498154Z INFO 07-23 02:49:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:54.932337177Z INFO 07-23 02:49:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:49:59.972912007Z INFO 07-23 02:49:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:05.007985393Z INFO 07-23 02:50:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:10.048282976Z INFO 07-23 02:50:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:15.051923596Z INFO 07-23 02:50:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.2%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:20.093869860Z INFO 07-23 02:50:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:25.125545341Z INFO 07-23 02:50:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:30.161930016Z INFO 07-23 02:50:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:35.204345197Z INFO 07-23 02:50:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:40.208651498Z INFO 07-23 02:50:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:45.211397957Z INFO 07-23 02:50:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:50.215514007Z INFO 07-23 02:50:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:50:55.226202583Z INFO 07-23 02:50:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:00.234263458Z INFO 07-23 02:51:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:05.268080221Z INFO 07-23 02:51:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:10.295027960Z INFO 07-23 02:51:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:15.333710639Z INFO 07-23 02:51:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:22.023882201Z INFO 07-23 02:51:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.1 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:32.024428280Z INFO 07-23 02:51:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:42.025051809Z INFO 07-23 02:51:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:51:52.025669180Z INFO 07-23 02:51:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:02.026245267Z INFO 07-23 02:52:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:12.025670280Z INFO 07-23 02:52:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:22.025991502Z INFO 07-23 02:52:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:32.026096441Z INFO 07-23 02:52:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:42.026445024Z INFO 07-23 02:52:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:52:52.026881786Z INFO 07-23 02:52:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:02.027443741Z INFO 07-23 02:53:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:12.027857295Z INFO 07-23 02:53:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:22.021717002Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.028015831Z INFO 07-23 02:53:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:22.034636325Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.042292538Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.049601298Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:22.237627231Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:53:27.059387000Z INFO 07-23 02:53:27 metrics.py:345] Avg prompt throughput: 1478.3 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:32.069675497Z INFO 07-23 02:53:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:37.075811510Z INFO 07-23 02:53:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:41.384843095Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:53:41.531870163Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:53:42.183280911Z INFO 07-23 02:53:42 metrics.py:345] Avg prompt throughput: 179.7 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:47.184870989Z INFO 07-23 02:53:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:52.202464890Z INFO 07-23 02:53:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:53:57.216053216Z INFO 07-23 02:53:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:02.236765351Z INFO 07-23 02:54:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:07.245326134Z INFO 07-23 02:54:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:12.257378000Z INFO 07-23 02:54:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:17.265723592Z INFO 07-23 02:54:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:22.274426444Z INFO 07-23 02:54:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:27.288180475Z INFO 07-23 02:54:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:32.303355245Z INFO 07-23 02:54:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:37.308955789Z INFO 07-23 02:54:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:42.316230767Z INFO 07-23 02:54:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:47.330731343Z INFO 07-23 02:54:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:52.350938716Z INFO 07-23 02:54:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:54:57.368955403Z INFO 07-23 02:54:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:02.387972119Z INFO 07-23 02:55:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:07.403238913Z INFO 07-23 02:55:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:12.424487516Z INFO 07-23 02:55:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:17.468209611Z INFO 07-23 02:55:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:22.468838506Z INFO 07-23 02:55:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:27.495305182Z INFO 07-23 02:55:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:31.663705730Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:55:33.559722894Z INFO 07-23 02:55:33 metrics.py:345] Avg prompt throughput: 675.4 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:39.017972246Z INFO 07-23 02:55:39 metrics.py:345] Avg prompt throughput: 2251.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:45.229906291Z INFO 07-23 02:55:45 metrics.py:345] Avg prompt throughput: 1978.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:52.031344888Z INFO 07-23 02:55:52 metrics.py:345] Avg prompt throughput: 1204.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:55:56.488143481Z (VllmWorkerProcess pid=346) INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:56.488439965Z INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:56.488465183Z (VllmWorkerProcess pid=345) INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:56.488468721Z (VllmWorkerProcess pid=344) INFO 07-23 02:55:56 corex_fa2.py:225] Using CoreX paged decode: B=1 Hq=4 Hkv=1 D=256 max_k=45455 partition=256
2026-07-23T02:55:57.074355971Z INFO 07-23 02:55:57 metrics.py:345] Avg prompt throughput: 1703.4 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:02.075744305Z INFO 07-23 02:56:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:04.960717007Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:05.133613133Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:07.092301075Z INFO 07-23 02:56:07 metrics.py:345] Avg prompt throughput: 664.6 tokens/s, Avg generation throughput: 12.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:12.126688361Z INFO 07-23 02:56:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:14.381995529Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.389331161Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.395208292Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.402701762Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:14.562022048Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:18.164082686Z INFO 07-23 02:56:18 metrics.py:345] Avg prompt throughput: 1356.9 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:23.187795865Z INFO 07-23 02:56:23 metrics.py:345] Avg prompt throughput: 620.6 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:28.216477560Z INFO 07-23 02:56:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:32.311129833Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:56:32.500362238Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:34.377488148Z INFO 07-23 02:56:34 metrics.py:345] Avg prompt throughput: 664.8 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:39.826490851Z INFO 07-23 02:56:39 metrics.py:345] Avg prompt throughput: 2255.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:44.855411293Z INFO 07-23 02:56:44 metrics.py:345] Avg prompt throughput: 1032.6 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:49.884515182Z INFO 07-23 02:56:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:56:54.030501607Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:56:55.903654883Z INFO 07-23 02:56:55 metrics.py:345] Avg prompt throughput: 680.5 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:01.492620867Z INFO 07-23 02:57:01 metrics.py:345] Avg prompt throughput: 1348.2 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:06.518263412Z INFO 07-23 02:57:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:06.756437999Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:57:12.034175081Z INFO 07-23 02:57:12 metrics.py:345] Avg prompt throughput: 1485.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:18.176599373Z INFO 07-23 02:57:18 metrics.py:345] Avg prompt throughput: 2667.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:24.913022642Z INFO 07-23 02:57:24 metrics.py:345] Avg prompt throughput: 1824.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:32.034111195Z INFO 07-23 02:57:32 metrics.py:345] Avg prompt throughput: 1150.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:42.034654469Z INFO 07-23 02:57:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:47.041274003Z INFO 07-23 02:57:47 metrics.py:345] Avg prompt throughput: 804.5 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:51.606342181Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:57:51.804362711Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:57:53.676250964Z INFO 07-23 02:57:53 metrics.py:345] Avg prompt throughput: 617.3 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:57:59.365803779Z INFO 07-23 02:57:59 metrics.py:345] Avg prompt throughput: 1703.3 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:04.393685506Z INFO 07-23 02:58:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:04.609577440Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:04.615677229Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:04.798268401Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:58:10.219376350Z INFO 07-23 02:58:10 metrics.py:345] Avg prompt throughput: 2109.3 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:15.231494329Z INFO 07-23 02:58:15 metrics.py:345] Avg prompt throughput: 266.8 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:16.102093564Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:16.116114529Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:16.122546146Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:58:16.358547897Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:58:21.798309841Z INFO 07-23 02:58:21 metrics.py:345] Avg prompt throughput: 1871.3 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:27.758005289Z INFO 07-23 02:58:27 metrics.py:345] Avg prompt throughput: 2061.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:34.479363188Z INFO 07-23 02:58:34 metrics.py:345] Avg prompt throughput: 1828.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:41.994543666Z INFO 07-23 02:58:41 metrics.py:345] Avg prompt throughput: 1635.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:46.793774434Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:58:48.658622582Z INFO 07-23 02:58:48 metrics.py:345] Avg prompt throughput: 631.3 tokens/s, Avg generation throughput: 11.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:53.685872484Z INFO 07-23 02:58:53 metrics.py:345] Avg prompt throughput: 764.6 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:58:58.727252608Z INFO 07-23 02:58:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:01.342048508Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T02:59:01.502531224Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:05.905542940Z INFO 07-23 02:59:05 metrics.py:345] Avg prompt throughput: 1007.1 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:10.916217422Z INFO 07-23 02:59:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:15.938840078Z INFO 07-23 02:59:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:16.919636112Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:22.003161588Z INFO 07-23 02:59:22 metrics.py:345] Avg prompt throughput: 1326.5 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:27.012505251Z INFO 07-23 02:59:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:31.000804350Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:32.036163466Z INFO 07-23 02:59:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:37.075964266Z INFO 07-23 02:59:37 metrics.py:345] Avg prompt throughput: 1428.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:42.093095363Z INFO 07-23 02:59:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:44.048812647Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T02:59:47.672900564Z INFO 07-23 02:59:47 metrics.py:345] Avg prompt throughput: 1468.2 tokens/s, Avg generation throughput: 6.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:53.382936907Z INFO 07-23 02:59:53 metrics.py:345] Avg prompt throughput: 2152.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T02:59:59.847651377Z INFO 07-23 02:59:59 metrics.py:345] Avg prompt throughput: 1900.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:12.037275698Z INFO 07-23 03:00:12 metrics.py:345] Avg prompt throughput: 336.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:17.039468718Z INFO 07-23 03:00:17 metrics.py:345] Avg prompt throughput: 728.1 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:20.892612495Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:00:22.757922531Z INFO 07-23 03:00:22 metrics.py:345] Avg prompt throughput: 716.3 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:27.762148011Z INFO 07-23 03:00:27 metrics.py:345] Avg prompt throughput: 990.8 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:32.765102431Z INFO 07-23 03:00:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:37.783726602Z INFO 07-23 03:00:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:41.716682007Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:00:43.588916994Z INFO 07-23 03:00:43 metrics.py:345] Avg prompt throughput: 705.6 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:48.604975727Z INFO 07-23 03:00:48 metrics.py:345] Avg prompt throughput: 877.8 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:53.636064513Z INFO 07-23 03:00:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:00:58.643811919Z INFO 07-23 03:00:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:03.437865787Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:01:04.011675373Z INFO 07-23 03:01:04 metrics.py:345] Avg prompt throughput: 134.5 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:09.032151825Z INFO 07-23 03:01:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:14.076641438Z INFO 07-23 03:01:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:19.116791927Z INFO 07-23 03:01:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:24.132816105Z INFO 07-23 03:01:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:29.152510624Z INFO 07-23 03:01:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:34.164408415Z INFO 07-23 03:01:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:39.177405529Z INFO 07-23 03:01:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:44.196112311Z INFO 07-23 03:01:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:49.218177392Z INFO 07-23 03:01:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:54.260843117Z INFO 07-23 03:01:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:01:59.263581709Z INFO 07-23 03:01:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:04.286227306Z INFO 07-23 03:02:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:09.307925835Z INFO 07-23 03:02:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:14.334136745Z INFO 07-23 03:02:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:19.371566476Z INFO 07-23 03:02:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:24.403118388Z INFO 07-23 03:02:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:29.442979636Z INFO 07-23 03:02:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:34.460345650Z INFO 07-23 03:02:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:34.943043684Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:02:39.543923430Z INFO 07-23 03:02:39 metrics.py:345] Avg prompt throughput: 1440.3 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:43.567655344Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:02:43.733772678Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:02:45.627591234Z INFO 07-23 03:02:45 metrics.py:345] Avg prompt throughput: 673.3 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:50.087651530Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:02:50.253227675Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:02:52.041399788Z INFO 07-23 03:02:52 metrics.py:345] Avg prompt throughput: 556.3 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:02:57.051443405Z INFO 07-23 03:02:57 metrics.py:345] Avg prompt throughput: 1924.7 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:02.093327354Z INFO 07-23 03:03:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:07.102889862Z INFO 07-23 03:03:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:07.595301247Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:12.607251038Z INFO 07-23 03:03:12 metrics.py:345] Avg prompt throughput: 1430.3 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:14.974811119Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:20.025072743Z INFO 07-23 03:03:20 metrics.py:345] Avg prompt throughput: 1075.1 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:22.619326919Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:26.117782285Z INFO 07-23 03:03:26 metrics.py:345] Avg prompt throughput: 1344.6 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:31.802449723Z INFO 07-23 03:03:31 metrics.py:345] Avg prompt throughput: 2161.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:40.400462617Z INFO 07-23 03:03:40 metrics.py:345] Avg prompt throughput: 786.3 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:40.860411102Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:03:46.589024290Z INFO 07-23 03:03:46 metrics.py:345] Avg prompt throughput: 1610.1 tokens/s, Avg generation throughput: 1.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:51.589997032Z INFO 07-23 03:03:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:56.633889073Z INFO 07-23 03:03:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:03:57.579527268Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:03:57.746631502Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:04:02.045624284Z INFO 07-23 03:04:02 metrics.py:345] Avg prompt throughput: 756.9 tokens/s, Avg generation throughput: 3.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:04.802148446Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:04:05.027520323Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:04:08.674733439Z INFO 07-23 03:04:08 metrics.py:345] Avg prompt throughput: 1849.3 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:14.390439518Z INFO 07-23 03:04:14 metrics.py:345] Avg prompt throughput: 2149.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:20.860969569Z INFO 07-23 03:04:20 metrics.py:345] Avg prompt throughput: 1899.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:28.091096635Z INFO 07-23 03:04:28 metrics.py:345] Avg prompt throughput: 1699.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:33.666207138Z INFO 07-23 03:04:33 metrics.py:345] Avg prompt throughput: 832.6 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:37.651979949Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:04:38.689261053Z INFO 07-23 03:04:38 metrics.py:345] Avg prompt throughput: 108.1 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:43.719316648Z INFO 07-23 03:04:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:48.726194909Z INFO 07-23 03:04:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:53.773772705Z INFO 07-23 03:04:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:04:58.783851267Z INFO 07-23 03:04:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:03.796039763Z INFO 07-23 03:05:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:08.800370314Z INFO 07-23 03:05:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:13.817925374Z INFO 07-23 03:05:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:18.827534721Z INFO 07-23 03:05:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:23.836345906Z INFO 07-23 03:05:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:28.867665562Z INFO 07-23 03:05:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:33.879535202Z INFO 07-23 03:05:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:38.887930697Z INFO 07-23 03:05:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:43.911415061Z INFO 07-23 03:05:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:48.919604312Z INFO 07-23 03:05:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:53.923735935Z INFO 07-23 03:05:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:05:58.937847984Z INFO 07-23 03:05:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:03.976626873Z INFO 07-23 03:06:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:08.988814054Z INFO 07-23 03:06:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:13.996690644Z INFO 07-23 03:06:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:19.022685216Z INFO 07-23 03:06:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:24.035035366Z INFO 07-23 03:06:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:29.045998728Z INFO 07-23 03:06:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:34.065591058Z INFO 07-23 03:06:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:39.065937087Z INFO 07-23 03:06:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:44.089850754Z INFO 07-23 03:06:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:49.122224783Z INFO 07-23 03:06:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:54.140258854Z INFO 07-23 03:06:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:06:59.155775804Z INFO 07-23 03:06:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:04.167641219Z INFO 07-23 03:07:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:09.183289842Z INFO 07-23 03:07:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:14.190459782Z INFO 07-23 03:07:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:16.063329774Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:07:19.693310580Z INFO 07-23 03:07:19 metrics.py:345] Avg prompt throughput: 1488.7 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:24.709589906Z INFO 07-23 03:07:24 metrics.py:345] Avg prompt throughput: 289.5 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:29.752714538Z INFO 07-23 03:07:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:34.754784357Z INFO 07-23 03:07:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:37.046291229Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:07:40.663717101Z INFO 07-23 03:07:40 metrics.py:345] Avg prompt throughput: 1386.4 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:45.698396136Z INFO 07-23 03:07:45 metrics.py:345] Avg prompt throughput: 1079.9 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:50.739722502Z INFO 07-23 03:07:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:55.759442021Z INFO 07-23 03:07:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:07:59.292075461Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:08:00.760756817Z INFO 07-23 03:08:00 metrics.py:345] Avg prompt throughput: 68.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:05.803486926Z INFO 07-23 03:08:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:10.807800210Z INFO 07-23 03:08:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:15.810120875Z INFO 07-23 03:08:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:20.821109568Z INFO 07-23 03:08:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:25.834563489Z INFO 07-23 03:08:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:30.852701551Z INFO 07-23 03:08:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:35.895279345Z INFO 07-23 03:08:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:40.898125953Z INFO 07-23 03:08:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:45.903238274Z INFO 07-23 03:08:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:50.911176380Z INFO 07-23 03:08:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:08:55.948442038Z INFO 07-23 03:08:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:00.986020640Z INFO 07-23 03:09:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:05.987833861Z INFO 07-23 03:09:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:10.999189231Z INFO 07-23 03:09:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:16.010005913Z INFO 07-23 03:09:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:21.023415174Z INFO 07-23 03:09:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:26.037961024Z INFO 07-23 03:09:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:31.059518782Z INFO 07-23 03:09:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:36.076762327Z INFO 07-23 03:09:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:41.098050215Z INFO 07-23 03:09:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:46.109931334Z INFO 07-23 03:09:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:51.117802613Z INFO 07-23 03:09:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:09:52.198541671Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:09:58.473656203Z INFO 07-23 03:09:58 metrics.py:345] Avg prompt throughput: 1424.5 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:03.483554227Z INFO 07-23 03:10:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:08.486025664Z INFO 07-23 03:10:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:13.387287348Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:10:13.573126997Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:15.451842926Z INFO 07-23 03:10:15 metrics.py:345] Avg prompt throughput: 588.0 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:20.484742816Z INFO 07-23 03:10:20 metrics.py:345] Avg prompt throughput: 1315.2 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:25.490834189Z INFO 07-23 03:10:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:30.496012765Z INFO 07-23 03:10:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:33.890420869Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:35.776362721Z INFO 07-23 03:10:35 metrics.py:345] Avg prompt throughput: 775.7 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:40.799476081Z INFO 07-23 03:10:40 metrics.py:345] Avg prompt throughput: 834.3 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:42.302078280Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:45.904718727Z INFO 07-23 03:10:45 metrics.py:345] Avg prompt throughput: 1604.6 tokens/s, Avg generation throughput: 5.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:50.918196398Z INFO 07-23 03:10:50 metrics.py:345] Avg prompt throughput: 43.5 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:10:51.861459428Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:10:51.868181568Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:10:52.095460779Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:10:57.543735685Z INFO 07-23 03:10:57 metrics.py:345] Avg prompt throughput: 1854.6 tokens/s, Avg generation throughput: 3.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:03.506535562Z INFO 07-23 03:11:03 metrics.py:345] Avg prompt throughput: 2060.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:10.241413771Z INFO 07-23 03:11:10 metrics.py:345] Avg prompt throughput: 1824.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:17.765981052Z INFO 07-23 03:11:17 metrics.py:345] Avg prompt throughput: 1633.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:22.772423832Z INFO 07-23 03:11:22 metrics.py:345] Avg prompt throughput: 173.2 tokens/s, Avg generation throughput: 3.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:27.820717941Z INFO 07-23 03:11:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:32.868761969Z INFO 07-23 03:11:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:35.822104096Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:11:39.432629061Z INFO 07-23 03:11:39 metrics.py:345] Avg prompt throughput: 1248.1 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:42.539380381Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:11:46.145813361Z INFO 07-23 03:11:46 metrics.py:345] Avg prompt throughput: 1271.2 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:50.849891660Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:50.859074820Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:50.870425879Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:51.098867253Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:11:52.776745737Z INFO 07-23 03:11:52 metrics.py:345] Avg prompt throughput: 71.2 tokens/s, Avg generation throughput: 12.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:57.791435469Z INFO 07-23 03:11:57 metrics.py:345] Avg prompt throughput: 1751.5 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:11:58.518847773Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:11:58.688843761Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:12:04.113810544Z INFO 07-23 03:12:04 metrics.py:345] Avg prompt throughput: 1943.6 tokens/s, Avg generation throughput: 2.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:09.124640595Z INFO 07-23 03:12:09 metrics.py:345] Avg prompt throughput: 450.4 tokens/s, Avg generation throughput: 7.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:14.096840115Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:12:14.320275730Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:12:16.080406340Z INFO 07-23 03:12:16 metrics.py:345] Avg prompt throughput: 588.9 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:21.516014892Z INFO 07-23 03:12:21 metrics.py:345] Avg prompt throughput: 2260.7 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:32.515215161Z INFO 07-23 03:12:32 metrics.py:345] Avg prompt throughput: 1016.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:33.019477144Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:12:41.016003493Z INFO 07-23 03:12:41 metrics.py:345] Avg prompt throughput: 1415.5 tokens/s, Avg generation throughput: 0.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:46.059272778Z INFO 07-23 03:12:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:51.061081047Z INFO 07-23 03:12:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:56.072726396Z INFO 07-23 03:12:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:12:58.413592732Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:13:02.114424022Z INFO 07-23 03:13:02 metrics.py:345] Avg prompt throughput: 1355.9 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:07.138412366Z INFO 07-23 03:13:07 metrics.py:345] Avg prompt throughput: 1180.3 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:10.703528957Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:10.723774197Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:10.959621972Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:13:12.777150696Z INFO 07-23 03:13:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:18.311467651Z INFO 07-23 03:13:18 metrics.py:345] Avg prompt throughput: 2960.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:24.528764448Z INFO 07-23 03:13:24 metrics.py:345] Avg prompt throughput: 1976.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:31.514201171Z INFO 07-23 03:13:31 metrics.py:345] Avg prompt throughput: 1759.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:42.028574197Z INFO 07-23 03:13:42 metrics.py:345] Avg prompt throughput: 277.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:45.431039577Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:45.437698822Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:45.444970744Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:13:45.625709408Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:13:47.511514890Z INFO 07-23 03:13:47 metrics.py:345] Avg prompt throughput: 747.0 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:52.776792073Z INFO 07-23 03:13:52 metrics.py:345] Avg prompt throughput: 777.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:13:57.800309946Z INFO 07-23 03:13:57 metrics.py:345] Avg prompt throughput: 690.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:02.809048544Z INFO 07-23 03:14:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:06.461208755Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.468193202Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.479204061Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.490777942Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:14:06.648810696Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:14:07.835638500Z INFO 07-23 03:14:07 metrics.py:345] Avg prompt throughput: 24.3 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:12.846371069Z INFO 07-23 03:14:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:17.850632021Z INFO 07-23 03:14:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:22.856280984Z INFO 07-23 03:14:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:27.888165813Z INFO 07-23 03:14:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:32.925170980Z INFO 07-23 03:14:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:37.936372523Z INFO 07-23 03:14:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:42.944534615Z INFO 07-23 03:14:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:47.945186825Z INFO 07-23 03:14:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:52.949910632Z INFO 07-23 03:14:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:14:57.964030722Z INFO 07-23 03:14:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:02.982507987Z INFO 07-23 03:15:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:07.998544641Z INFO 07-23 03:15:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:13.004333260Z INFO 07-23 03:15:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:18.016318554Z INFO 07-23 03:15:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:23.029382219Z INFO 07-23 03:15:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:28.041986622Z INFO 07-23 03:15:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:33.065708355Z INFO 07-23 03:15:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:38.078323753Z INFO 07-23 03:15:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:43.091103774Z INFO 07-23 03:15:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:48.108899253Z INFO 07-23 03:15:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:53.135039995Z INFO 07-23 03:15:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:15:58.165410734Z INFO 07-23 03:15:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:03.183514389Z INFO 07-23 03:16:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:08.205135732Z INFO 07-23 03:16:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:13.216887456Z INFO 07-23 03:16:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:18.224956585Z INFO 07-23 03:16:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:20.572927902Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:16:20.746589986Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:16:23.267047992Z INFO 07-23 03:16:23 metrics.py:345] Avg prompt throughput: 107.7 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:28.282762875Z INFO 07-23 03:16:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:33.296712243Z INFO 07-23 03:16:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:38.326019630Z INFO 07-23 03:16:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:43.357754792Z INFO 07-23 03:16:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:48.372927578Z INFO 07-23 03:16:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:53.390454474Z INFO 07-23 03:16:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:16:58.399988303Z INFO 07-23 03:16:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:03.405932485Z INFO 07-23 03:17:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:08.446484958Z INFO 07-23 03:17:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:13.485751050Z INFO 07-23 03:17:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:18.523721424Z INFO 07-23 03:17:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:23.560268144Z INFO 07-23 03:17:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:28.594709480Z INFO 07-23 03:17:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:33.630996999Z INFO 07-23 03:17:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:38.670563497Z INFO 07-23 03:17:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:43.672950199Z INFO 07-23 03:17:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:48.687444828Z INFO 07-23 03:17:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:53.687871665Z INFO 07-23 03:17:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:17:58.699681518Z INFO 07-23 03:17:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:03.710114472Z INFO 07-23 03:18:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:08.720217881Z INFO 07-23 03:18:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:13.743844160Z INFO 07-23 03:18:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:18.764123061Z INFO 07-23 03:18:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:23.778691605Z INFO 07-23 03:18:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:28.790043878Z INFO 07-23 03:18:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:33.801253262Z INFO 07-23 03:18:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:38.807935708Z INFO 07-23 03:18:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:43.815867459Z INFO 07-23 03:18:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:48.827006448Z INFO 07-23 03:18:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:53.834181643Z INFO 07-23 03:18:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:18:58.865607506Z INFO 07-23 03:18:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:03.879818981Z INFO 07-23 03:19:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:08.900000692Z INFO 07-23 03:19:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:10.466463776Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:10.478993493Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:10.679321643Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:19:14.309575673Z INFO 07-23 03:19:14 metrics.py:345] Avg prompt throughput: 1514.4 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:19.322028134Z INFO 07-23 03:19:19 metrics.py:345] Avg prompt throughput: 972.8 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:24.355050263Z INFO 07-23 03:19:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:27.145073359Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:19:31.802504328Z INFO 07-23 03:19:31 metrics.py:345] Avg prompt throughput: 1008.4 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:36.308014528Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:36.316129768Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:19:36.507397695Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:19:38.394733836Z INFO 07-23 03:19:38 metrics.py:345] Avg prompt throughput: 621.3 tokens/s, Avg generation throughput: 15.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:43.661430418Z INFO 07-23 03:19:43 metrics.py:345] Avg prompt throughput: 1761.8 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:48.702576396Z INFO 07-23 03:19:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:53.735280536Z INFO 07-23 03:19:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:19:55.502354844Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:20:00.322746405Z INFO 07-23 03:20:00 metrics.py:345] Avg prompt throughput: 1168.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:02.888919108Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:20:02.897148113Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:20:03.074751698Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:20:06.698577320Z INFO 07-23 03:20:06 metrics.py:345] Avg prompt throughput: 1284.8 tokens/s, Avg generation throughput: 9.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:11.953751610Z INFO 07-23 03:20:11 metrics.py:345] Avg prompt throughput: 1225.3 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:16.958423659Z INFO 07-23 03:20:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:21.958724991Z INFO 07-23 03:20:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:26.962080264Z INFO 07-23 03:20:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:31.188353870Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:20:31.346968400Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:20:31.986096383Z INFO 07-23 03:20:31 metrics.py:345] Avg prompt throughput: 65.5 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:36.991956897Z INFO 07-23 03:20:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:42.022761244Z INFO 07-23 03:20:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:47.065410115Z INFO 07-23 03:20:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:52.071533963Z INFO 07-23 03:20:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:20:57.095045358Z INFO 07-23 03:20:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:02.100354460Z INFO 07-23 03:21:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:07.120462554Z INFO 07-23 03:21:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:12.144278143Z INFO 07-23 03:21:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:17.160746225Z INFO 07-23 03:21:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:22.183849444Z INFO 07-23 03:21:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:27.201981662Z INFO 07-23 03:21:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:32.222207283Z INFO 07-23 03:21:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:37.226790110Z INFO 07-23 03:21:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:42.263162787Z INFO 07-23 03:21:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:47.285217965Z INFO 07-23 03:21:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:52.294860803Z INFO 07-23 03:21:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:21:57.305607847Z INFO 07-23 03:21:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:02.332677933Z INFO 07-23 03:22:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:07.361450993Z INFO 07-23 03:22:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:12.386346091Z INFO 07-23 03:22:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:17.425448234Z INFO 07-23 03:22:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:22.445119521Z INFO 07-23 03:22:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:27.467709607Z INFO 07-23 03:22:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:32.488520263Z INFO 07-23 03:22:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:37.510941058Z INFO 07-23 03:22:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:42.548234171Z INFO 07-23 03:22:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:47.581833368Z INFO 07-23 03:22:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:52.623687606Z INFO 07-23 03:22:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:22:57.640445190Z INFO 07-23 03:22:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:02.662738298Z INFO 07-23 03:23:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:04.568763829Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:23:04.740545928Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:23:07.680005320Z INFO 07-23 03:23:07 metrics.py:345] Avg prompt throughput: 228.6 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:12.697581987Z INFO 07-23 03:23:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:17.717154268Z INFO 07-23 03:23:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:22.737304950Z INFO 07-23 03:23:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:27.757062716Z INFO 07-23 03:23:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:32.761707602Z INFO 07-23 03:23:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:37.779115679Z INFO 07-23 03:23:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:42.787757479Z INFO 07-23 03:23:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:47.821253928Z INFO 07-23 03:23:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:52.843645340Z INFO 07-23 03:23:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:23:57.888462280Z INFO 07-23 03:23:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:02.888897990Z INFO 07-23 03:24:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:07.905867858Z INFO 07-23 03:24:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:12.928719262Z INFO 07-23 03:24:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:17.947424275Z INFO 07-23 03:24:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:22.973047988Z INFO 07-23 03:24:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:27.993404362Z INFO 07-23 03:24:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:33.038567398Z INFO 07-23 03:24:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:38.072799979Z INFO 07-23 03:24:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:43.107134360Z INFO 07-23 03:24:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:48.139340196Z INFO 07-23 03:24:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:53.158864324Z INFO 07-23 03:24:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:24:58.184946563Z INFO 07-23 03:24:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:03.205227792Z INFO 07-23 03:25:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:08.227341986Z INFO 07-23 03:25:08 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:13.266499819Z INFO 07-23 03:25:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:18.273249978Z INFO 07-23 03:25:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:23.310387392Z INFO 07-23 03:25:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:28.347385529Z INFO 07-23 03:25:28 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:33.372635744Z INFO 07-23 03:25:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:38.389145564Z INFO 07-23 03:25:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:43.404210116Z INFO 07-23 03:25:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:48.431958800Z INFO 07-23 03:25:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:53.458237948Z INFO 07-23 03:25:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:25:58.483877167Z INFO 07-23 03:25:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:03.522336181Z INFO 07-23 03:26:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:05.090021844Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:05.095951361Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:05.295502128Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:26:08.926311950Z INFO 07-23 03:26:08 metrics.py:345] Avg prompt throughput: 1516.0 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:14.842702111Z INFO 07-23 03:26:14 metrics.py:345] Avg prompt throughput: 1164.9 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:19.852487057Z INFO 07-23 03:26:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:20.202580536Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:20.210313004Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:20.384973960Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:26:24.999260358Z INFO 07-23 03:26:24 metrics.py:345] Avg prompt throughput: 1444.2 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:30.041607883Z INFO 07-23 03:26:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:35.048754058Z INFO 07-23 03:26:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:39.100631558Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:39.109347200Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:26:39.304823096Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:26:41.188117195Z INFO 07-23 03:26:41 metrics.py:345] Avg prompt throughput: 667.2 tokens/s, Avg generation throughput: 15.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:50.433459245Z INFO 07-23 03:26:50 metrics.py:345] Avg prompt throughput: 1319.3 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:26:55.457907052Z INFO 07-23 03:26:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:00.476478203Z INFO 07-23 03:27:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:01.588214832Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:27:07.025171015Z INFO 07-23 03:27:07 metrics.py:345] Avg prompt throughput: 1876.4 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:12.629720324Z INFO 07-23 03:27:12 metrics.py:345] Avg prompt throughput: 1105.2 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:17.668027044Z INFO 07-23 03:27:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:22.673085829Z INFO 07-23 03:27:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:27.691946553Z INFO 07-23 03:27:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:29.151414399Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:27:29.743848827Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:27:29.763701013Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:27:29.917975218Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:27:32.793393663Z INFO 07-23 03:27:32 metrics.py:345] Avg prompt throughput: 802.9 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:37.836214318Z INFO 07-23 03:27:37 metrics.py:345] Avg prompt throughput: 708.7 tokens/s, Avg generation throughput: 14.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:42.859098255Z INFO 07-23 03:27:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:44.256741458Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:27:47.875184669Z INFO 07-23 03:27:47 metrics.py:345] Avg prompt throughput: 1633.2 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:52.899691834Z INFO 07-23 03:27:52 metrics.py:345] Avg prompt throughput: 155.6 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:27:57.919289532Z INFO 07-23 03:27:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:02.702062753Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:28:02.873611630Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:28:03.112144515Z INFO 07-23 03:28:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:08.150500129Z INFO 07-23 03:28:08 metrics.py:345] Avg prompt throughput: 1578.7 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:13.187908126Z INFO 07-23 03:28:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:17.164320870Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:28:17.327688086Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:28:19.210550904Z INFO 07-23 03:28:19 metrics.py:345] Avg prompt throughput: 680.1 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:24.225292173Z INFO 07-23 03:28:24 metrics.py:345] Avg prompt throughput: 800.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:29.226764995Z INFO 07-23 03:28:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:31.869759250Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:28:34.263598007Z INFO 07-23 03:28:34 metrics.py:345] Avg prompt throughput: 132.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:39.292111567Z INFO 07-23 03:28:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:44.324452648Z INFO 07-23 03:28:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:49.348232646Z INFO 07-23 03:28:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:54.355931285Z INFO 07-23 03:28:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:28:59.363795090Z INFO 07-23 03:28:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:04.392516862Z INFO 07-23 03:29:04 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:09.415443309Z INFO 07-23 03:29:09 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:14.436739524Z INFO 07-23 03:29:14 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:19.457395216Z INFO 07-23 03:29:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:24.474718167Z INFO 07-23 03:29:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:29.501523520Z INFO 07-23 03:29:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:34.523233969Z INFO 07-23 03:29:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:39.552660459Z INFO 07-23 03:29:39 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:44.570080552Z INFO 07-23 03:29:44 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:49.596454253Z INFO 07-23 03:29:49 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:54.615672916Z INFO 07-23 03:29:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:29:59.633932098Z INFO 07-23 03:29:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:02.589700120Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:02.778803816Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:04.677928907Z INFO 07-23 03:30:04 metrics.py:345] Avg prompt throughput: 812.1 tokens/s, Avg generation throughput: 13.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:09.720978113Z INFO 07-23 03:30:09 metrics.py:345] Avg prompt throughput: 952.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:11.253121488Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:14.873610618Z INFO 07-23 03:30:14 metrics.py:345] Avg prompt throughput: 1589.9 tokens/s, Avg generation throughput: 5.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:21.905152531Z INFO 07-23 03:30:21 metrics.py:345] Avg prompt throughput: 1420.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:26.927246350Z INFO 07-23 03:30:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:26.949212854Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:26.957860631Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:26.966418697Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:27.141252545Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:31.939042290Z INFO 07-23 03:30:31 metrics.py:345] Avg prompt throughput: 1712.4 tokens/s, Avg generation throughput: 2.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:36.949470944Z INFO 07-23 03:30:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:40.533602444Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:40.691497271Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:41.010347463Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:41.018689226Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:41.178014687Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:43.067206037Z INFO 07-23 03:30:43 metrics.py:345] Avg prompt throughput: 685.9 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:48.085902193Z INFO 07-23 03:30:48 metrics.py:345] Avg prompt throughput: 617.9 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:53.113753547Z INFO 07-23 03:30:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:30:53.864374472Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:53.875008951Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:30:54.049288150Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:30:58.801164893Z INFO 07-23 03:30:58 metrics.py:345] Avg prompt throughput: 1346.1 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:01.135308567Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:31:06.029877137Z INFO 07-23 03:31:06 metrics.py:345] Avg prompt throughput: 1074.5 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:08.592483770Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:31:08.960823866Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:31:12.640101178Z INFO 07-23 03:31:12 metrics.py:345] Avg prompt throughput: 1239.3 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:18.383940479Z INFO 07-23 03:31:18 metrics.py:345] Avg prompt throughput: 2139.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:24.879158343Z INFO 07-23 03:31:24 metrics.py:345] Avg prompt throughput: 1891.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:32.155291335Z INFO 07-23 03:31:32 metrics.py:345] Avg prompt throughput: 1688.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:37.443808749Z INFO 07-23 03:31:37 metrics.py:345] Avg prompt throughput: 1549.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:43.111724266Z INFO 07-23 03:31:43 metrics.py:345] Avg prompt throughput: 1445.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:49.164283827Z INFO 07-23 03:31:49 metrics.py:345] Avg prompt throughput: 1353.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:31:55.618757782Z INFO 07-23 03:31:55 metrics.py:345] Avg prompt throughput: 1269.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:02.450252476Z INFO 07-23 03:32:02 metrics.py:345] Avg prompt throughput: 1199.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:09.683638726Z INFO 07-23 03:32:09 metrics.py:345] Avg prompt throughput: 1132.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:17.325293296Z INFO 07-23 03:32:17 metrics.py:345] Avg prompt throughput: 1072.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:23.119540943Z INFO 07-23 03:32:23 metrics.py:345] Avg prompt throughput: 706.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:33.119590916Z INFO 07-23 03:32:33 metrics.py:345] Avg prompt throughput: 409.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:43.120993751Z INFO 07-23 03:32:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:51.139668086Z INFO 07-23 03:32:51 metrics.py:345] Avg prompt throughput: 349.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:32:56.182102278Z INFO 07-23 03:32:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:01.222431844Z INFO 07-23 03:33:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:06.255444585Z INFO 07-23 03:33:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:11.294163817Z INFO 07-23 03:33:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:16.330648490Z INFO 07-23 03:33:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:21.366805857Z INFO 07-23 03:33:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:26.401669203Z INFO 07-23 03:33:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:31.427649231Z INFO 07-23 03:33:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:36.462316305Z INFO 07-23 03:33:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:41.488932088Z INFO 07-23 03:33:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:46.518394807Z INFO 07-23 03:33:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:51.547857990Z INFO 07-23 03:33:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:33:56.580255579Z INFO 07-23 03:33:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:01.580987774Z INFO 07-23 03:34:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:06.616073565Z INFO 07-23 03:34:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:11.650597177Z INFO 07-23 03:34:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:16.654305379Z INFO 07-23 03:34:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:21.683189679Z INFO 07-23 03:34:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:26.707783194Z INFO 07-23 03:34:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:31.739851567Z INFO 07-23 03:34:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:36.764364675Z INFO 07-23 03:34:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:41.785282509Z INFO 07-23 03:34:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:46.807385509Z INFO 07-23 03:34:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:51.832113797Z INFO 07-23 03:34:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:34:56.852758637Z INFO 07-23 03:34:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:01.877195030Z INFO 07-23 03:35:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:06.909124996Z INFO 07-23 03:35:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:11.936863441Z INFO 07-23 03:35:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:16.969294745Z INFO 07-23 03:35:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:22.001685150Z INFO 07-23 03:35:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:27.035357579Z INFO 07-23 03:35:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:32.070780441Z INFO 07-23 03:35:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:37.109800889Z INFO 07-23 03:35:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:42.144817317Z INFO 07-23 03:35:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:47.182019980Z INFO 07-23 03:35:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:52.225007586Z INFO 07-23 03:35:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:35:57.259046506Z INFO 07-23 03:35:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:02.297575559Z INFO 07-23 03:36:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:07.340493043Z INFO 07-23 03:36:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:12.378520548Z INFO 07-23 03:36:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:17.409447662Z INFO 07-23 03:36:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:22.439119517Z INFO 07-23 03:36:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:27.442704506Z INFO 07-23 03:36:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:32.480948868Z INFO 07-23 03:36:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:37.529398347Z INFO 07-23 03:36:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:42.574099184Z INFO 07-23 03:36:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:47.612483559Z INFO 07-23 03:36:47 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:52.652951855Z INFO 07-23 03:36:52 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:36:57.687904692Z INFO 07-23 03:36:57 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:02.726593377Z INFO 07-23 03:37:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:07.730913728Z INFO 07-23 03:37:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:12.774245347Z INFO 07-23 03:37:12 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:17.813872698Z INFO 07-23 03:37:17 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:22.848066921Z INFO 07-23 03:37:22 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:27.882473147Z INFO 07-23 03:37:27 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:32.917401272Z INFO 07-23 03:37:32 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:37.932243776Z INFO 07-23 03:37:37 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:42.963779527Z INFO 07-23 03:37:42 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:48.007317544Z INFO 07-23 03:37:48 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:53.048938883Z INFO 07-23 03:37:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:58.085090714Z INFO 07-23 03:37:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 38.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:37:59.337134045Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:03.126573524Z INFO 07-23 03:38:03 metrics.py:345] Avg prompt throughput: 1624.9 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:07.915203230Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:38:07.925057068Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:38:08.094963074Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:09.982564809Z INFO 07-23 03:38:09 metrics.py:345] Avg prompt throughput: 715.7 tokens/s, Avg generation throughput: 12.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:14.987905774Z INFO 07-23 03:38:14 metrics.py:345] Avg prompt throughput: 1116.8 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:19.993703211Z INFO 07-23 03:38:19 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:23.734983446Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:25.623769661Z INFO 07-23 03:38:25 metrics.py:345] Avg prompt throughput: 727.5 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:30.635619226Z INFO 07-23 03:38:30 metrics.py:345] Avg prompt throughput: 1021.6 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:34.343758361Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:38:34.566766277Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:36.464600680Z INFO 07-23 03:38:36 metrics.py:345] Avg prompt throughput: 702.7 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:41.913518197Z INFO 07-23 03:38:41 metrics.py:345] Avg prompt throughput: 2255.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:53.127802320Z INFO 07-23 03:38:53 metrics.py:345] Avg prompt throughput: 730.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:38:57.626063560Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:38:59.523823531Z INFO 07-23 03:38:59 metrics.py:345] Avg prompt throughput: 1128.4 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:04.973947484Z INFO 07-23 03:39:04 metrics.py:345] Avg prompt throughput: 2254.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:13.127817976Z INFO 07-23 03:39:13 metrics.py:345] Avg prompt throughput: 502.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:18.136657170Z INFO 07-23 03:39:18 metrics.py:345] Avg prompt throughput: 697.6 tokens/s, Avg generation throughput: 16.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:19.550416132Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:39:23.976378415Z INFO 07-23 03:39:23 metrics.py:345] Avg prompt throughput: 1237.9 tokens/s, Avg generation throughput: 4.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:29.002950812Z INFO 07-23 03:39:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:34.041860533Z INFO 07-23 03:39:34 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:34.620637687Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:39:41.578722845Z INFO 07-23 03:39:41 metrics.py:345] Avg prompt throughput: 1490.8 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:46.612817940Z INFO 07-23 03:39:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:51.651629828Z INFO 07-23 03:39:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:56.654212752Z INFO 07-23 03:39:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:39:58.976469155Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:40:02.603723546Z INFO 07-23 03:40:02 metrics.py:345] Avg prompt throughput: 1376.9 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:08.305651965Z INFO 07-23 03:40:08 metrics.py:345] Avg prompt throughput: 2155.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:16.236480761Z INFO 07-23 03:40:16 metrics.py:345] Avg prompt throughput: 478.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:21.267816676Z INFO 07-23 03:40:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:26.295602231Z INFO 07-23 03:40:26 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:28.778018078Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:40:32.408324583Z INFO 07-23 03:40:32 metrics.py:345] Avg prompt throughput: 1340.2 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:38.108997575Z INFO 07-23 03:40:38 metrics.py:345] Avg prompt throughput: 2155.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:43.129334599Z INFO 07-23 03:40:43 metrics.py:345] Avg prompt throughput: 1631.7 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:48.160479129Z INFO 07-23 03:40:48 metrics.py:345] Avg prompt throughput: 120.6 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:53.164874777Z INFO 07-23 03:40:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:58.169636135Z INFO 07-23 03:40:58 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:40:59.795739240Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:41:03.414206853Z INFO 07-23 03:41:03 metrics.py:345] Avg prompt throughput: 1562.0 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:08.455624617Z INFO 07-23 03:41:08 metrics.py:345] Avg prompt throughput: 728.7 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:13.456923903Z INFO 07-23 03:41:13 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:18.485525514Z INFO 07-23 03:41:18 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:23.498236924Z INFO 07-23 03:41:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:26.268303762Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:41:31.070539610Z INFO 07-23 03:41:31 metrics.py:345] Avg prompt throughput: 1015.4 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:36.096941176Z INFO 07-23 03:41:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:41.130726930Z INFO 07-23 03:41:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:41.559287741Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:41:41.969406342Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:41:47.474648078Z INFO 07-23 03:41:47 metrics.py:345] Avg prompt throughput: 1937.0 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:41:53.131561420Z INFO 07-23 03:41:53 metrics.py:345] Avg prompt throughput: 1448.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:00.176943964Z INFO 07-23 03:42:00 metrics.py:345] Avg prompt throughput: 2325.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:07.700147555Z INFO 07-23 03:42:07 metrics.py:345] Avg prompt throughput: 1633.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:13.132480415Z INFO 07-23 03:42:13 metrics.py:345] Avg prompt throughput: 754.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:19.035572074Z INFO 07-23 03:42:19 metrics.py:345] Avg prompt throughput: 2081.6 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:33.133304474Z INFO 07-23 03:42:33 metrics.py:345] Avg prompt throughput: 290.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:43.134438022Z INFO 07-23 03:42:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:48.154183452Z INFO 07-23 03:42:48 metrics.py:345] Avg prompt throughput: 735.3 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:53.191208919Z INFO 07-23 03:42:53 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:42:57.644819022Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:42:59.536790168Z INFO 07-23 03:42:59 metrics.py:345] Avg prompt throughput: 645.5 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:04.984151481Z INFO 07-23 03:43:04 metrics.py:345] Avg prompt throughput: 2255.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:13.134871432Z INFO 07-23 03:43:13 metrics.py:345] Avg prompt throughput: 1005.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:18.371162432Z INFO 07-23 03:43:18 metrics.py:345] Avg prompt throughput: 757.4 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:23.343108547Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:43:23.588052243Z INFO 07-23 03:43:23 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:28.848071558Z INFO 07-23 03:43:28 metrics.py:345] Avg prompt throughput: 2336.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:34.822293294Z INFO 07-23 03:43:34 metrics.py:345] Avg prompt throughput: 2056.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:41.559053064Z INFO 07-23 03:43:41 metrics.py:345] Avg prompt throughput: 1824.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:42.134554199Z INFO: 172.28.22.23:54024 - "GET /v1/models HTTP/1.1" 200 OK
2026-07-23T03:43:49.093692247Z INFO 07-23 03:43:49 metrics.py:345] Avg prompt throughput: 1630.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:43:54.573982630Z INFO 07-23 03:43:54 metrics.py:345] Avg prompt throughput: 1494.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:00.419430557Z INFO 07-23 03:44:00 metrics.py:345] Avg prompt throughput: 1401.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:06.652667894Z INFO 07-23 03:44:06 metrics.py:345] Avg prompt throughput: 1314.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:13.273466906Z INFO 07-23 03:44:13 metrics.py:345] Avg prompt throughput: 1237.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:23.589944720Z INFO 07-23 03:44:23 metrics.py:345] Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:33.589622516Z INFO 07-23 03:44:33 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:40.373070879Z INFO 07-23 03:44:40 metrics.py:345] Avg prompt throughput: 477.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:45.378779262Z INFO 07-23 03:44:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:49.409870456Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:44:51.294134680Z INFO 07-23 03:44:51 metrics.py:345] Avg prompt throughput: 692.4 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:44:57.327566458Z INFO 07-23 03:44:57 metrics.py:345] Avg prompt throughput: 1304.6 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:02.343004242Z INFO 07-23 03:45:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:07.353037375Z INFO 07-23 03:45:07 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:08.703038876Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:45:08.945392988Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:45:12.589865901Z INFO 07-23 03:45:12 metrics.py:345] Avg prompt throughput: 1564.3 tokens/s, Avg generation throughput: 5.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:18.308064429Z INFO 07-23 03:45:18 metrics.py:345] Avg prompt throughput: 2148.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:23.590532919Z INFO 07-23 03:45:23 metrics.py:345] Avg prompt throughput: 1550.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:29.508225758Z INFO 07-23 03:45:29 metrics.py:345] Avg prompt throughput: 2076.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:40.164335600Z INFO 07-23 03:45:40 metrics.py:345] Avg prompt throughput: 286.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:45.215691270Z INFO 07-23 03:45:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:50.262555523Z INFO 07-23 03:45:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:45:55.305684193Z INFO 07-23 03:45:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:00.329990814Z INFO 07-23 03:46:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:05.358893348Z INFO 07-23 03:46:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:10.388523949Z INFO 07-23 03:46:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:15.420504664Z INFO 07-23 03:46:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:20.448707580Z INFO 07-23 03:46:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:25.464481774Z INFO 07-23 03:46:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:30.502872201Z INFO 07-23 03:46:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:35.530078168Z INFO 07-23 03:46:35 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:40.559825975Z INFO 07-23 03:46:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:45.595368097Z INFO 07-23 03:46:45 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:50.632083164Z INFO 07-23 03:46:50 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:46:55.670123461Z INFO 07-23 03:46:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:00.692064975Z INFO 07-23 03:47:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:05.731449900Z INFO 07-23 03:47:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:10.775862113Z INFO 07-23 03:47:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:15.789807502Z INFO 07-23 03:47:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:18.030344386Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:47:18.267069376Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:47:21.908553681Z INFO 07-23 03:47:21 metrics.py:345] Avg prompt throughput: 1338.8 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:27.607452205Z INFO 07-23 03:47:27 metrics.py:345] Avg prompt throughput: 2156.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:33.592167279Z INFO 07-23 03:47:33 metrics.py:345] Avg prompt throughput: 684.4 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:38.615627195Z INFO 07-23 03:47:38 metrics.py:345] Avg prompt throughput: 610.3 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:43.652542974Z INFO 07-23 03:47:43 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:48.069171000Z ERROR 07-23 03:47:48 serving_chat.py:221] Error in applying chat template from request
2026-07-23T03:47:48.069190665Z ERROR 07-23 03:47:48 serving_chat.py:221] Traceback (most recent call last):
2026-07-23T03:47:48.069193533Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 188, in create_chat_completion
2026-07-23T03:47:48.069196058Z ERROR 07-23 03:47:48 serving_chat.py:221] conversation, mm_data_future = parse_chat_messages_futures(
2026-07-23T03:47:48.069198382Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 551, in parse_chat_messages_futures
2026-07-23T03:47:48.069200645Z ERROR 07-23 03:47:48 serving_chat.py:221] sub_messages = _parse_chat_message_content(msg, mm_tracker)
2026-07-23T03:47:48.069202837Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 477, in _parse_chat_message_content
2026-07-23T03:47:48.069204889Z ERROR 07-23 03:47:48 serving_chat.py:221] result = _parse_chat_message_content_parts(
2026-07-23T03:47:48.069206940Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 429, in _parse_chat_message_content_parts
2026-07-23T03:47:48.069209541Z ERROR 07-23 03:47:48 serving_chat.py:221] mm_parser.parse_image(image_url["url"])
2026-07-23T03:47:48.069211572Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 311, in parse_image
2026-07-23T03:47:48.069213819Z ERROR 07-23 03:47:48 serving_chat.py:221] placeholder = self._tracker.add("image", image_coro)
2026-07-23T03:47:48.069215830Z ERROR 07-23 03:47:48 serving_chat.py:221] File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/chat_utils.py", line 220, in add
2026-07-23T03:47:48.069218119Z ERROR 07-23 03:47:48 serving_chat.py:221] raise ValueError(
2026-07-23T03:47:48.069229290Z ERROR 07-23 03:47:48 serving_chat.py:221] ValueError: At most 1 image(s) may be provided in one request.
2026-07-23T03:47:48.069293988Z /usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py:222: RuntimeWarning: coroutine 'async_get_and_parse_image' was never awaited
2026-07-23T03:47:48.069300137Z return self.create_error_response(str(e))
2026-07-23T03:47:48.069302341Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback
2026-07-23T03:47:48.069481495Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:47:48.325722518Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:47:50.228821656Z INFO 07-23 03:47:50 metrics.py:345] Avg prompt throughput: 622.9 tokens/s, Avg generation throughput: 14.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:47:55.685485411Z INFO 07-23 03:47:55 metrics.py:345] Avg prompt throughput: 2251.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:01.889056918Z INFO 07-23 03:48:01 metrics.py:345] Avg prompt throughput: 1980.8 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:08.880652390Z INFO 07-23 03:48:08 metrics.py:345] Avg prompt throughput: 1757.5 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:20.950602402Z INFO 07-23 03:48:20 metrics.py:345] Avg prompt throughput: 547.1 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:25.952959584Z INFO 07-23 03:48:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:30.999396546Z INFO 07-23 03:48:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:36.044256010Z INFO 07-23 03:48:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:41.044599828Z INFO 07-23 03:48:41 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:46.083073696Z INFO 07-23 03:48:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:51.124521665Z INFO 07-23 03:48:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2026-07-23T03:48:56.168277560Z INFO 07-23 03:48:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:01.212516850Z INFO 07-23 03:49:01 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:06.258871180Z INFO 07-23 03:49:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:11.303607987Z INFO 07-23 03:49:11 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:16.352861160Z INFO 07-23 03:49:16 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:21.355919111Z INFO 07-23 03:49:21 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:23.979457774Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:49:27.605136815Z INFO 07-23 03:49:27 metrics.py:345] Avg prompt throughput: 1310.9 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:33.296245524Z INFO 07-23 03:49:33 metrics.py:345] Avg prompt throughput: 2159.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:38.656394179Z INFO 07-23 03:49:38 metrics.py:345] Avg prompt throughput: 1597.3 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:41.259500869Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:49:44.876448503Z INFO 07-23 03:49:44 metrics.py:345] Avg prompt throughput: 1317.1 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:50.779280791Z INFO 07-23 03:49:50 metrics.py:345] Avg prompt throughput: 1163.2 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:49:55.782459543Z INFO 07-23 03:49:55 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:00.821850285Z INFO 07-23 03:50:00 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:05.860302968Z INFO 07-23 03:50:05 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:10.896836932Z INFO 07-23 03:50:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:15.907192499Z INFO 07-23 03:50:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.1%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:19.131493471Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:50:21.026655055Z INFO 07-23 03:50:21 metrics.py:345] Avg prompt throughput: 800.1 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:26.482545766Z INFO 07-23 03:50:26 metrics.py:345] Avg prompt throughput: 2252.2 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:32.675695952Z INFO 07-23 03:50:32 metrics.py:345] Avg prompt throughput: 1984.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:37.717894719Z INFO 07-23 03:50:37 metrics.py:345] Avg prompt throughput: 104.5 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:38.747799235Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:50:43.596272691Z INFO 07-23 03:50:43 metrics.py:345] Avg prompt throughput: 1393.6 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:49.494533413Z INFO 07-23 03:50:49 metrics.py:345] Avg prompt throughput: 1317.8 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:54.495926964Z INFO 07-23 03:50:54 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%.
2026-07-23T03:50:59.532585217Z INFO 07-23 03:50:59 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.3%, CPU KV cache usage: 0.0%.
2026-07-23T03:51:04.243701758Z INFO: 172.28.22.23:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-23T03:51:04.695557111Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, Traceback (most recent call last):
2026-07-23T03:51:04.695590033Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
2026-07-23T03:51:04.695593398Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
2026-07-23T03:51:04.695597304Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695599822Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695601960Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 69, in start_worker_execution_loop
2026-07-23T03:51:04.695604422Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.execute_model(execute_model_req=None)
2026-07-23T03:51:04.695606461Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.695608857Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.model_runner.execute_model(
2026-07-23T03:51:04.695611175Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695613359Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695615304Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.695617660Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.695619640Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695621844Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695623828Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695628609Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695630599Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.695632812Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states = self.model(
2026-07-23T03:51:04.695634801Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695636954Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695639175Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695641326Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695643268Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.695645430Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = layer(
2026-07-23T03:51:04.695647355Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695649500Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695651500Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695653916Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695655924Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.695658142Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.695660265Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695662484Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695667765Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695670920Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695672956Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.695675139Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.695677985Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.695680320Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_native(x, residual)
2026-07-23T03:51:04.695684136Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.695686306Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.695688422Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 158, in forward_static
2026-07-23T03:51:04.695690647Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] variance = x.pow(2).mean(dim=-1, keepdim=True)
2026-07-23T03:51:04.695693383Z (VllmWorkerProcess pid=345) (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 1 has a total capacty of 31.72 GiB of which 74.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, Traceback (most recent call last):
2026-07-23T03:51:04.695697049Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
2026-07-23T03:51:04.695699616Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
2026-07-23T03:51:04.695701640Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695706279Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695708636Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 69, in start_worker_execution_loop
2026-07-23T03:51:04.695710796Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.execute_model(execute_model_req=None)
2026-07-23T03:51:04.695712766Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.695714957Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] output = self.model_runner.execute_model(
2026-07-23T03:51:04.695716932Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.695719271Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return func(*args, **kwargs)
2026-07-23T03:51:04.695721336Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.695723498Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.695725472Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695727695Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695729775Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695732069Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695734030Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.695736157Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states = self.model(
2026-07-23T03:51:04.695738121Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695742683Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695744681Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695746918Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695748963Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.695751447Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = layer(
2026-07-23T03:51:04.695753660Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695755775Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695757874Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695759993Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695762002Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.695764118Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.695766114Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.695768209Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.695770214Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.695772460Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.695774624Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.695778783Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.695781237Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.695783998Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_native(x, residual)
2026-07-23T03:51:04.695785906Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.695788026Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.695790208Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 158, in forward_static
2026-07-23T03:51:04.695792433Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] variance = x.pow(2).mean(dim=-1, keepdim=True)
2026-07-23T03:51:04.695794820Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.695797659Z (VllmWorkerProcess pid=345) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231]
2026-07-23T03:51:04.695799657Z ERROR 07-23 03:51:04 multiproc_worker_utils.py:231] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 1 has a total capacty of 31.72 GiB of which 74.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.695802893Z (VllmWorkerProcess pid=344) ERROR 07-23 03:51:04 multiproc_worker_utils.py:231]
2026-07-23T03:51:04.699701286Z ERROR 07-23 03:51:04 async_llm_engine.py:66] Engine background task failed
2026-07-23T03:51:04.699707282Z ERROR 07-23 03:51:04 async_llm_engine.py:66] Traceback (most recent call last):
2026-07-23T03:51:04.699709690Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 56, in _log_task_completion
2026-07-23T03:51:04.699712553Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return_value = task.result()
2026-07-23T03:51:04.699722044Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 857, in run_engine_loop
2026-07-23T03:51:04.699724371Z ERROR 07-23 03:51:04 async_llm_engine.py:66] result = task.result()
2026-07-23T03:51:04.699726357Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 776, in engine_step
2026-07-23T03:51:04.699728437Z ERROR 07-23 03:51:04 async_llm_engine.py:66] request_outputs = await self.engine.step_async(virtual_engine)
2026-07-23T03:51:04.699730549Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 348, in step_async
2026-07-23T03:51:04.699732927Z ERROR 07-23 03:51:04 async_llm_engine.py:66] outputs = await self.model_executor.execute_model_async(
2026-07-23T03:51:04.699734814Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 181, in execute_model_async
2026-07-23T03:51:04.699736873Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return await self._driver_execute_model_async(execute_model_req)
2026-07-23T03:51:04.699738752Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 224, in _driver_execute_model_async
2026-07-23T03:51:04.699740932Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return await self.driver_exec_model(execute_model_req)
2026-07-23T03:51:04.699743070Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2026-07-23T03:51:04.699745042Z ERROR 07-23 03:51:04 async_llm_engine.py:66] result = self.fn(*self.args, **self.kwargs)
2026-07-23T03:51:04.699747316Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.699749564Z ERROR 07-23 03:51:04 async_llm_engine.py:66] output = self.model_runner.execute_model(
2026-07-23T03:51:04.699751539Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.699753508Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return func(*args, **kwargs)
2026-07-23T03:51:04.699755607Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.699758036Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.699760153Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699762173Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699764016Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699768920Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699770911Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.699772926Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_states = self.model(
2026-07-23T03:51:04.699774758Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699776744Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699778565Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699780536Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699782347Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.699784609Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_states, residual = layer(
2026-07-23T03:51:04.699786447Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699788829Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699790628Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699792704Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699856811Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.699859662Z ERROR 07-23 03:51:04 async_llm_engine.py:66] hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.699861913Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.699864092Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.699866211Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.699868266Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return forward_call(*args, **kwargs)
2026-07-23T03:51:04.699870390Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.699872478Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.699876920Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.699879423Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self.forward_native(x, residual)
2026-07-23T03:51:04.699881751Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.699884537Z ERROR 07-23 03:51:04 async_llm_engine.py:66] return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.699886523Z ERROR 07-23 03:51:04 async_llm_engine.py:66] File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_static
2026-07-23T03:51:04.699888605Z ERROR 07-23 03:51:04 async_llm_engine.py:66] x = x.float()
2026-07-23T03:51:04.699890682Z ERROR 07-23 03:51:04 async_llm_engine.py:66] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.700138219Z Exception in callback functools.partial(<function _log_task_completion at 0x7f206fedf1c0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f206034f010>>)
2026-07-23T03:51:04.700142915Z handle: <Handle functools.partial(<function _log_task_completion at 0x7f206fedf1c0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f206034f010>>)>
2026-07-23T03:51:04.700146529Z Traceback (most recent call last):
2026-07-23T03:51:04.700148465Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 56, in _log_task_completion
2026-07-23T03:51:04.700150727Z return_value = task.result()
2026-07-23T03:51:04.700152569Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 857, in run_engine_loop
2026-07-23T03:51:04.700154521Z result = task.result()
2026-07-23T03:51:04.700156285Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 776, in engine_step
2026-07-23T03:51:04.700158380Z request_outputs = await self.engine.step_async(virtual_engine)
2026-07-23T03:51:04.700160278Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 348, in step_async
2026-07-23T03:51:04.700162359Z outputs = await self.model_executor.execute_model_async(
2026-07-23T03:51:04.700164171Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 181, in execute_model_async
2026-07-23T03:51:04.700166399Z return await self._driver_execute_model_async(execute_model_req)
2026-07-23T03:51:04.700170872Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 224, in _driver_execute_model_async
2026-07-23T03:51:04.700172919Z return await self.driver_exec_model(execute_model_req)
2026-07-23T03:51:04.700174704Z File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2026-07-23T03:51:04.700176814Z result = self.fn(*self.args, **self.kwargs)
2026-07-23T03:51:04.700178633Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:04.700180563Z output = self.model_runner.execute_model(
2026-07-23T03:51:04.700182330Z File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:04.700184286Z return func(*args, **kwargs)
2026-07-23T03:51:04.700186064Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:04.700188266Z hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:04.700190129Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700192031Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700193904Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700195807Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700197556Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:04.700199508Z hidden_states = self.model(
2026-07-23T03:51:04.700201263Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700203336Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700205463Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700207334Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700209080Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:04.700211063Z hidden_states, residual = layer(
2026-07-23T03:51:04.700212851Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700214768Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700217261Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700219719Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700221463Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:04.700223405Z hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:04.700228613Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:04.700230566Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:04.700232462Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:04.700234414Z return forward_call(*args, **kwargs)
2026-07-23T03:51:04.700236143Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:04.700238046Z return self._forward_method(*args, **kwargs)
2026-07-23T03:51:04.700239806Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:04.700241998Z return self.forward_native(x, residual)
2026-07-23T03:51:04.700243823Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:04.700245757Z return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:04.700247627Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_static
2026-07-23T03:51:04.700249543Z x = x.float()
2026-07-23T03:51:04.700252832Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:04.700256610Z
2026-07-23T03:51:04.700258408Z The above exception was the direct cause of the following exception:
2026-07-23T03:51:04.700260214Z
2026-07-23T03:51:04.700261917Z Traceback (most recent call last):
2026-07-23T03:51:04.700263623Z File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
2026-07-23T03:51:04.700265499Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 68, in _log_task_completion
2026-07-23T03:51:04.700268134Z raise AsyncEngineDeadError(
2026-07-23T03:51:04.700269863Z vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.
2026-07-23T03:51:04.702552963Z ERROR: Exception in ASGI application
2026-07-23T03:51:04.702558866Z Traceback (most recent call last):
2026-07-23T03:51:04.702560834Z File "/usr/local/lib/python3.10/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
2026-07-23T03:51:04.702564379Z yield
2026-07-23T03:51:04.702566507Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 271, in __call__
2026-07-23T03:51:04.702571357Z async with anyio.create_task_group() as task_group:
2026-07-23T03:51:04.702573227Z File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
2026-07-23T03:51:04.702575469Z raise BaseExceptionGroup(
2026-07-23T03:51:04.702577231Z exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2026-07-23T03:51:04.702579140Z
2026-07-23T03:51:04.702580984Z During handling of the above exception, another exception occurred:
2026-07-23T03:51:04.702582850Z
2026-07-23T03:51:04.702584544Z Traceback (most recent call last):
2026-07-23T03:51:04.702586535Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-07-23T03:51:04.702588429Z await app(scope, receive, sender)
2026-07-23T03:51:04.702590235Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
2026-07-23T03:51:04.702592118Z await response(scope, receive, send)
2026-07-23T03:51:04.702593877Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 270, in __call__
2026-07-23T03:51:04.702595840Z with collapse_excgroups():
2026-07-23T03:51:04.702597561Z File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
2026-07-23T03:51:04.702599394Z self.gen.throw(typ, value, traceback)
2026-07-23T03:51:04.702601159Z File "/usr/local/lib/python3.10/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
2026-07-23T03:51:04.702603068Z raise exc
2026-07-23T03:51:04.702604874Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 274, in wrap
2026-07-23T03:51:04.702606737Z await func()
2026-07-23T03:51:04.702608502Z File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 254, in stream_response
2026-07-23T03:51:04.702610428Z async for chunk in self.body_iterator:
2026-07-23T03:51:04.702612210Z File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 846, in chat_completion_stream_generator
2026-07-23T03:51:04.702614205Z await self.engine_client.abort(request_id)
2026-07-23T03:51:04.702615921Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 1243, in abort
2026-07-23T03:51:04.702617838Z raise AsyncEngineDeadError(
2026-07-23T03:51:04.702619831Z vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is not running. If it was running, inspect the output to find the stacktrace of the error that caused the background loop to stop (AsyncEngineDeadError).
2026-07-23T03:51:04.702622281Z
2026-07-23T03:51:04.702623917Z The above exception was the direct cause of the following exception:
2026-07-23T03:51:04.702625718Z
2026-07-23T03:51:04.702627946Z Traceback (most recent call last):
2026-07-23T03:51:04.702643211Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-07-23T03:51:04.702645150Z await app(scope, receive, sender)
2026-07-23T03:51:04.702646977Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 716, in __call__
2026-07-23T03:51:04.702648835Z await self.middleware_stack(scope, receive, send)
2026-07-23T03:51:04.702650643Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 736, in app
2026-07-23T03:51:04.702652476Z await route.handle(scope, receive, send)
2026-07-23T03:51:04.702654253Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 290, in handle
2026-07-23T03:51:04.702656095Z await self.app(scope, receive, send)
2026-07-23T03:51:04.702657830Z File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 78, in app
2026-07-23T03:51:04.702659880Z await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2026-07-23T03:51:04.702662244Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 56, in wrapped_app
2026-07-23T03:51:04.702665694Z raise RuntimeError("Caught handled exception, but response already started.") from exc
2026-07-23T03:51:04.702667893Z RuntimeError: Caught handled exception, but response already started.
2026-07-23T03:51:04.702669731Z
2026-07-23T03:51:04.702671340Z The above exception was the direct cause of the following exception:
2026-07-23T03:51:04.702673239Z
2026-07-23T03:51:04.702674862Z Traceback (most recent call last):
2026-07-23T03:51:04.702677233Z File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
2026-07-23T03:51:04.702679161Z result = await app( # type: ignore[func-returns-value]
2026-07-23T03:51:04.702680952Z File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2026-07-23T03:51:04.702682811Z return await self.app(scope, receive, send)
2026-07-23T03:51:04.702684557Z File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1082, in __call__
2026-07-23T03:51:04.702686691Z await super().__call__(scope, receive, send)
2026-07-23T03:51:04.702688431Z File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
2026-07-23T03:51:04.702690476Z await self.middleware_stack(scope, receive, send)
2026-07-23T03:51:04.702692252Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
2026-07-23T03:51:04.702694354Z raise exc
2026-07-23T03:51:04.702696382Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
2026-07-23T03:51:04.702698525Z await self.app(scope, receive, _send)
2026-07-23T03:51:04.702700285Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
2026-07-23T03:51:04.702705835Z await self.app(scope, receive, send)
2026-07-23T03:51:04.702708304Z File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
2026-07-23T03:51:04.702710205Z await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2026-07-23T03:51:04.702712202Z File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 56, in wrapped_app
2026-07-23T03:51:04.702714127Z raise RuntimeError("Caught handled exception, but response already started.") from exc
2026-07-23T03:51:04.702716055Z RuntimeError: Caught handled exception, but response already started.
2026-07-23T03:51:04.713173377Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.714970623Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.718637314Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.718694821Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.722498257Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.722585244Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.735233268Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:51:04.756590608Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:51:04.768186558Z CRITICAL 07-23 03:51:04 launcher.py:88] AsyncLLMEngine is already dead, terminating server process
2026-07-23T03:51:04.768268921Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-07-23T03:51:04.781736020Z INFO: 172.28.22.23:61225 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2026-07-23T03:51:04.791948299Z INFO: Shutting down
2026-07-23T03:51:04.892200042Z INFO: Waiting for application shutdown.
2026-07-23T03:51:04.892346436Z INFO: Application shutdown complete.
2026-07-23T03:51:04.893243235Z INFO: Finished server process [1]
2026-07-23T03:51:06.253648359Z INFO 07-23 03:51:06 multiproc_worker_utils.py:121] Killing local vLLM worker processes
2026-07-23T03:51:10.732959761Z Future exception was never retrieved
2026-07-23T03:51:10.732970009Z future: <Future finished exception=OutOfMemoryError('CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')>
2026-07-23T03:51:10.732975335Z Traceback (most recent call last):
2026-07-23T03:51:10.732983365Z File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 466, in chat_completion_stream_generator
2026-07-23T03:51:10.732985745Z async for res in result_generator:
2026-07-23T03:51:10.732987633Z File "/usr/local/corex/lib/python3/dist-packages/vllm/utils.py", line 458, in iterate_with_cancellation
2026-07-23T03:51:10.732989646Z item = await awaits[0]
2026-07-23T03:51:10.732991495Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 1046, in generate
2026-07-23T03:51:10.732993455Z async for output in await self.add_request(
2026-07-23T03:51:10.732995398Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 114, in generator
2026-07-23T03:51:10.732997331Z raise result
2026-07-23T03:51:10.732999159Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 56, in _log_task_completion
2026-07-23T03:51:10.733001089Z return_value = task.result()
2026-07-23T03:51:10.733002909Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 857, in run_engine_loop
2026-07-23T03:51:10.733004845Z result = task.result()
2026-07-23T03:51:10.733006732Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 776, in engine_step
2026-07-23T03:51:10.733008659Z request_outputs = await self.engine.step_async(virtual_engine)
2026-07-23T03:51:10.733010626Z File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 348, in step_async
2026-07-23T03:51:10.733012560Z outputs = await self.model_executor.execute_model_async(
2026-07-23T03:51:10.733014593Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 181, in execute_model_async
2026-07-23T03:51:10.733016583Z return await self._driver_execute_model_async(execute_model_req)
2026-07-23T03:51:10.733018783Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 224, in _driver_execute_model_async
2026-07-23T03:51:10.733020797Z return await self.driver_exec_model(execute_model_req)
2026-07-23T03:51:10.733022570Z File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2026-07-23T03:51:10.733024502Z result = self.fn(*self.args, **self.kwargs)
2026-07-23T03:51:10.733026262Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
2026-07-23T03:51:10.733028655Z output = self.model_runner.execute_model(
2026-07-23T03:51:10.733030678Z File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2026-07-23T03:51:10.733032715Z return func(*args, **kwargs)
2026-07-23T03:51:10.733034659Z File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1679, in execute_model
2026-07-23T03:51:10.733039337Z hidden_or_intermediate_states = model_executable(
2026-07-23T03:51:10.733041153Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733043068Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733044905Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733046777Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733050243Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1421, in forward
2026-07-23T03:51:10.733053412Z hidden_states = self.model(
2026-07-23T03:51:10.733057551Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733059640Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733061352Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733063252Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733065023Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1165, in forward
2026-07-23T03:51:10.733067165Z hidden_states, residual = layer(
2026-07-23T03:51:10.733069092Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733071000Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733072984Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733074867Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733076607Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/models/qwen3_5.py", line 1086, in forward
2026-07-23T03:51:10.733078748Z hidden_states, residual = self.post_attention_layernorm(
2026-07-23T03:51:10.733080781Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2026-07-23T03:51:10.733082662Z return self._call_impl(*args, **kwargs)
2026-07-23T03:51:10.733084425Z File "/usr/local/corex/lib/python3/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2026-07-23T03:51:10.733086295Z return forward_call(*args, **kwargs)
2026-07-23T03:51:10.733088148Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/custom_op.py", line 16, in forward
2026-07-23T03:51:10.733090040Z return self._forward_method(*args, **kwargs)
2026-07-23T03:51:10.733091807Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 187, in forward_cuda
2026-07-23T03:51:10.733093814Z return self.forward_native(x, residual)
2026-07-23T03:51:10.733095539Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 172, in forward_native
2026-07-23T03:51:10.733101307Z return self.forward_static(self.weight.data, self.variance_epsilon, x,
2026-07-23T03:51:10.733104565Z File "/usr/local/corex/lib/python3/dist-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_static
2026-07-23T03:51:10.733107808Z x = x.float()
2026-07-23T03:51:10.733109873Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 31.72 GiB of which 50.92 MiB is free. Of the allocated memory 30.86 GiB is allocated by PyTorch, and 210.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2026-07-23T03:51:10.735532351Z Task exception was never retrieved
2026-07-23T03:51:10.735545848Z future: <Task finished name='Task-129921' coro=<MultiprocessingGPUExecutorAsync._start_worker_execution_loop() done, defined at /usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py:253> exception=OutOfMemoryError('CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')>
2026-07-23T03:51:10.735550322Z Traceback (most recent call last):
2026-07-23T03:51:10.735552351Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 258, in _start_worker_execution_loop
2026-07-23T03:51:10.735554626Z return await asyncio.gather(*coros)
2026-07-23T03:51:10.735556397Z File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 183, in execute_method_async
2026-07-23T03:51:10.735558462Z return await future
2026-07-23T03:51:10.735560338Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacty of 31.72 GiB of which 70.92 MiB is free. Of the allocated memory 30.89 GiB is allocated by PyTorch, and 210.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Reference in New Issue View Git Blame Copy Permalink