document multi-step scheduler diagnostics
This commit is contained in:
@@ -268,3 +268,22 @@
|
|||||||
- Increasing concurrency does not improve aggregate decode throughput and worsens per-request TPS.
|
- Increasing concurrency does not improve aggregate decode throughput and worsens per-request TPS.
|
||||||
- Low average GPU utilization suggests scheduling, TP synchronization, MoE kernel, paged attention, or xFormers backend limitations before raw compute saturation.
|
- Low average GPU utilization suggests scheduling, TP synchronization, MoE kernel, paged attention, or xFormers backend limitations before raw compute saturation.
|
||||||
- Next focused experiment: disable chunked prefill and retry `--num-scheduler-steps 4` as a decode-only diagnostic variant.
|
- Next focused experiment: disable chunked prefill and retry `--num-scheduler-steps 4` as a decode-only diagnostic variant.
|
||||||
|
|
||||||
|
## 2026-07-14 Multi-Step Scheduling Diagnostic
|
||||||
|
|
||||||
|
- Goal: verify whether `--num-scheduler-steps 4` can improve decode throughput.
|
||||||
|
- Appended details to:
|
||||||
|
- `worklogs/decode_analysis_report_2026-07-14.md`
|
||||||
|
- Synced remote failure logs to:
|
||||||
|
- `worklogs/remote_results/2026-07-14-scheduler/`
|
||||||
|
- Experiments:
|
||||||
|
- No chunked prefill + `max_model_len=100000` failed because `max_num_batched_tokens=8192` is smaller than `max_model_len=100000`.
|
||||||
|
- No chunked prefill + `max_model_len=4096` still failed because `Multi-Step not supported for attention backend: xformers`.
|
||||||
|
- Forced `VLLM_ATTENTION_BACKEND=FLASHINFER` failed because `BatchDecodeWithPagedKVCacheWrapper` was `None`, indicating FlashInfer dependencies/wrappers are unavailable or incompatible.
|
||||||
|
- Conclusion:
|
||||||
|
- Multi-step scheduling cannot currently be tested or used on this server because the available attention backend is xFormers.
|
||||||
|
- The next practical direction is not more `num_scheduler_steps` tuning, but either making FlashAttention/FlashInfer backend available or profiling the current xFormers + MoE + TP decode path.
|
||||||
|
- Recovery:
|
||||||
|
- Restored full parser service with chunked prefill and prefix cache.
|
||||||
|
- Recovery log: `/root/work/logs/server_exp_seq2_b8192_fullparser_restored_after_schedtest.log`.
|
||||||
|
- `/health` returned `200`.
|
||||||
|
|||||||
@@ -297,3 +297,137 @@ GPU 利用率和功耗都偏低。虽然瞬时 GPU-Util 能到 100%，但平均
|
|||||||
4. 若没有提升，进入 qwen3_moe / fused_moe / attention 的代码级 profiling
|
4. 若没有提升，进入 qwen3_moe / fused_moe / attention 的代码级 profiling
|
||||||
|
|
||||||
本轮最重要的事实是：GPU 平均利用率只有约 `29%`，所以先不要把问题简单归因为“卡算不动”。更像是当前 decode 执行路径没有把四张卡持续喂满。
|
本轮最重要的事实是：GPU 平均利用率只有约 `29%`，所以先不要把问题简单归因为“卡算不动”。更像是当前 decode 执行路径没有把四张卡持续喂满。
|
||||||
|
|
||||||
|
## 七、调度方向验证实验：multi-step decode
|
||||||
|
|
||||||
|
追加日期：2026-07-14
|
||||||
|
|
||||||
|
### 目标
|
||||||
|
|
||||||
|
验证 `--num-scheduler-steps 4` 是否能作为 decode 吞吐提升方向。
|
||||||
|
|
||||||
|
由于上一轮实验显示并发升高没有带来 aggregate TPS 增益，且平均 GPU 利用率只有约 `29%`，multi-step scheduling 是最直接的调度侧候选优化：它理论上可以减少每 token 调度往返和 Python/worker 协调开销，让 decode 连续执行多个 step。
|
||||||
|
|
||||||
|
### 实验 1：关闭 chunked prefill，直接启用 multi-step
|
||||||
|
|
||||||
|
启动变体：
|
||||||
|
|
||||||
|
- `--num-scheduler-steps 4`
|
||||||
|
- 不加 `--enable-chunked-prefill`
|
||||||
|
- `--max-model-len 100000`
|
||||||
|
- `--max-num-batched-tokens 8192`
|
||||||
|
- 其它参数沿用完整 parser 服务配置
|
||||||
|
|
||||||
|
结果：
|
||||||
|
|
||||||
|
- 服务未启动。
|
||||||
|
- 远端日志：`/root/work/logs/server_exp_sched4_nochunk_fullparser.log`
|
||||||
|
- 本地归档：`worklogs/remote_results/2026-07-14-scheduler/server_exp_sched4_nochunk_fullparser.log`
|
||||||
|
|
||||||
|
失败原因：
|
||||||
|
|
||||||
|
```text
|
||||||
|
ValueError: max_num_batched_tokens (8192) is smaller than max_model_len (100000).
|
||||||
|
```
|
||||||
|
|
||||||
|
解释：
|
||||||
|
|
||||||
|
关闭 chunked prefill 后，vLLM 要求 `max_num_batched_tokens >= max_model_len`，否则实际最大可处理序列会被 `max_num_batched_tokens` 限制。这个配置不能用于官方长上下文，也无法进入 decode 压测。
|
||||||
|
|
||||||
|
### 实验 2：decode-only 诊断配置
|
||||||
|
|
||||||
|
为绕过实验 1 的限制，临时降低上下文长度，只用于短 prompt decode 诊断。
|
||||||
|
|
||||||
|
启动变体：
|
||||||
|
|
||||||
|
- `--max-model-len 4096`
|
||||||
|
- `--max-seq-len-to-capture 4096`
|
||||||
|
- `--max-num-batched-tokens 8192`
|
||||||
|
- `--num-scheduler-steps 4`
|
||||||
|
- 不加 `--enable-chunked-prefill`
|
||||||
|
- attention backend 仍为自动选择
|
||||||
|
|
||||||
|
结果：
|
||||||
|
|
||||||
|
- 服务未启动。
|
||||||
|
- 远端日志：`/root/work/logs/server_exp_sched4_nochunk_len4096_fullparser.log`
|
||||||
|
- 本地归档：`worklogs/remote_results/2026-07-14-scheduler/server_exp_sched4_nochunk_len4096_fullparser.log`
|
||||||
|
|
||||||
|
关键日志：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Using XFormers backend.
|
||||||
|
ValueError: Multi-Step not supported for attention backend: xformers.
|
||||||
|
Set VLLM_ATTENTION_BACKEND to a value from ['flash-attn', 'rocm-flash-attn', 'flashinfer'].
|
||||||
|
```
|
||||||
|
|
||||||
|
解释：
|
||||||
|
|
||||||
|
这说明当前环境不是“chunked prefill 与 multi-step 的组合不支持”这么简单，而是 `xformers` attention backend 本身不支持 multi-step worker。只要 attention backend 仍然落到 xFormers，multi-step 调度就无法启用。
|
||||||
|
|
||||||
|
### 实验 3：强制 FlashInfer backend
|
||||||
|
|
||||||
|
启动变体：
|
||||||
|
|
||||||
|
- 环境变量：`VLLM_ATTENTION_BACKEND=FLASHINFER`
|
||||||
|
- `--max-model-len 4096`
|
||||||
|
- `--num-scheduler-steps 4`
|
||||||
|
- 不加 `--enable-chunked-prefill`
|
||||||
|
- 其它参数同实验 2
|
||||||
|
|
||||||
|
结果：
|
||||||
|
|
||||||
|
- 服务未启动。
|
||||||
|
- 远端日志：`/root/work/logs/server_exp_sched4_nochunk_len4096_flashinfer.log`
|
||||||
|
- 本地归档：`worklogs/remote_results/2026-07-14-scheduler/server_exp_sched4_nochunk_len4096_flashinfer.log`
|
||||||
|
|
||||||
|
关键日志：
|
||||||
|
|
||||||
|
```text
|
||||||
|
TypeError: 'NoneType' object is not callable
|
||||||
|
...
|
||||||
|
self._decode_wrapper = BatchDecodeWithPagedKVCacheWrapper(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
解释：
|
||||||
|
|
||||||
|
`vllm/attention/backends/flashinfer.py` 会导入：
|
||||||
|
|
||||||
|
- `flashinfer.BatchDecodeWithPagedKVCacheWrapper`
|
||||||
|
- `flashinfer.decode.CUDAGraphBatchDecodeWithPagedKVCacheWrapper`
|
||||||
|
- `flashinfer.prefill.BatchPrefillWithPagedKVCacheWrapper`
|
||||||
|
- `ixformer.contrib.vllm_flash_attn.flash_attn_varlen_func`
|
||||||
|
|
||||||
|
当前环境中至少有关键 FlashInfer wrapper 没导入成功，导致 wrapper 为 `None`，在 profiling 阶段调用时报错。
|
||||||
|
|
||||||
|
### 实验结论
|
||||||
|
|
||||||
|
本轮没有进入 c1/c2/c4 decode 曲线压测，因为 multi-step 服务在启动阶段就失败。
|
||||||
|
|
||||||
|
结论不是“调度一定无效”，而是：
|
||||||
|
|
||||||
|
1. 当前 xFormers backend 下，multi-step 调度不可用。
|
||||||
|
2. 当前 FlashInfer backend 依赖不完整或与 CoreX 环境不兼容，不能直接替代 xFormers。
|
||||||
|
3. 自动 FlashAttention 也不可用；此前日志已显示 `vllm_flash_attn` 包缺失，因此自动回落到 xFormers。
|
||||||
|
|
||||||
|
因此，调度优化如果要继续推进，前置任务是 attention backend 适配：
|
||||||
|
|
||||||
|
- 路线 A：补齐/修复 CoreX 环境里的 FlashAttention 或 FlashInfer backend。
|
||||||
|
- 路线 B：改造 xFormers backend 或 MultiStepModelRunner，使其支持当前 xFormers 路径。
|
||||||
|
- 路线 C：绕过 multi-step，直接 profile xFormers decode、paged attention、MoE 与 TP 通信开销。
|
||||||
|
|
||||||
|
### 对下一步方向的影响
|
||||||
|
|
||||||
|
短期内，继续调 `--num-scheduler-steps` 没意义；它被 backend 卡住了。
|
||||||
|
|
||||||
|
下一步建议改为两条线并行：
|
||||||
|
|
||||||
|
1. **backend 可用性线**
|
||||||
|
- 检查 `flashinfer` 和 `ixformer.contrib.vllm_flash_attn` 在服务器上的实际导入错误。
|
||||||
|
- 确认官方镜像/包中是否本应包含 `vllm_flash_attn`。
|
||||||
|
- 如果能补齐依赖，再重跑 multi-step decode 曲线。
|
||||||
|
|
||||||
|
2. **代码 profiling 线**
|
||||||
|
- 直接在当前可用 xFormers 路径插桩。
|
||||||
|
- 重点记录每 token decode 中 attention、MoE、sampler、TP 同步的耗时。
|
||||||
|
- 当前平均 GPU 利用率低，profiling 比继续盲调参数更有价值。
|
||||||
|
|||||||
@@ -0,0 +1,41 @@
|
|||||||
|
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
|
||||||
|
import pynvml # type: ignore[import]
|
||||||
|
INFO 07-14 09:19:28 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
|
||||||
|
2026-07-14 09:19:30.064377: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
|
||||||
|
2026-07-14 09:19:30.115922: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
|
||||||
|
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
|
||||||
|
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
|
||||||
|
INFO 07-14 09:19:35 api_server.py:530] vLLM API server version 0.6.3
|
||||||
|
INFO 07-14 09:19:35 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=32768, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=4, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
|
||||||
|
INFO 07-14 09:19:35 config.py:1670] Downcasting torch.float32 to torch.float16.
|
||||||
|
INFO 07-14 09:19:46 config.py:887] Defaulting to use mp for distributed inference
|
||||||
|
WARNING 07-14 09:19:46 arg_utils.py:963] The model has a long context length (100000). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
|
||||||
|
return _run_code(code, main_globals, None,
|
||||||
|
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
|
||||||
|
exec(code, run_globals)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 595, in <module>
|
||||||
|
uvloop.run(run_server(args))
|
||||||
|
File "/usr/local/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
|
||||||
|
return loop.run_until_complete(wrapper())
|
||||||
|
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
|
||||||
|
File "/usr/local/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
|
||||||
|
return await main
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 562, in run_server
|
||||||
|
async with build_async_engine_client(args) as engine_client:
|
||||||
|
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
|
||||||
|
return await anext(self.gen)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 108, in build_async_engine_client
|
||||||
|
async with build_async_engine_client_from_engine_args(
|
||||||
|
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
|
||||||
|
return await anext(self.gen)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 130, in build_async_engine_client_from_engine_args
|
||||||
|
engine_config = engine_args.create_engine_config()
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/arg_utils.py", line 1021, in create_engine_config
|
||||||
|
scheduler_config = SchedulerConfig(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/config.py", line 1021, in __init__
|
||||||
|
self._verify_args()
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/config.py", line 1026, in _verify_args
|
||||||
|
raise ValueError(
|
||||||
|
ValueError: max_num_batched_tokens (8192) is smaller than max_model_len (100000). This effectively limits the maximum sequence length to max_num_batched_tokens and makes vLLM reject longer sequences. Please increase max_num_batched_tokens or decrease max_model_len.
|
||||||
@@ -0,0 +1,212 @@
|
|||||||
|
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
|
||||||
|
import pynvml # type: ignore[import]
|
||||||
|
INFO 07-14 09:43:16 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
|
||||||
|
2026-07-14 09:43:17.917732: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
|
||||||
|
2026-07-14 09:43:17.968433: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
|
||||||
|
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
|
||||||
|
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
|
||||||
|
INFO 07-14 09:43:23 api_server.py:530] vLLM API server version 0.6.3
|
||||||
|
INFO 07-14 09:43:23 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=4096, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=4096, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=4, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
|
||||||
|
INFO 07-14 09:43:23 config.py:1670] Downcasting torch.float32 to torch.float16.
|
||||||
|
INFO 07-14 09:43:34 config.py:887] Defaulting to use mp for distributed inference
|
||||||
|
WARNING 07-14 09:43:34 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
|
||||||
|
INFO 07-14 09:43:34 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=4, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
|
||||||
|
INFO 07-14 09:43:34 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
|
||||||
|
INFO 07-14 09:43:34 selector.py:141] Using Flashinfer backend.
|
||||||
|
WARNING 07-14 09:43:34 registry.py:205] `mm_limits` has already been set for model=/root/public-storage/models/Qwen/Qwen3.6-35B-A3B, and will be overwritten by the new values.
|
||||||
|
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
|
||||||
|
import pynvml # type: ignore[import]
|
||||||
|
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
|
||||||
|
import pynvml # type: ignore[import]
|
||||||
|
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
|
||||||
|
import pynvml # type: ignore[import]
|
||||||
|
INFO 07-14 09:43:36 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
|
||||||
|
INFO 07-14 09:43:36 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
|
||||||
|
INFO 07-14 09:43:36 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
|
||||||
|
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
|
||||||
|
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
|
||||||
|
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m INFO 07-14 09:43:44 selector.py:141] Using Flashinfer backend.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m WARNING 07-14 09:43:44 registry.py:205] `mm_limits` has already been set for model=/root/public-storage/models/Qwen/Qwen3.6-35B-A3B, and will be overwritten by the new values.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m INFO 07-14 09:43:44 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m INFO 07-14 09:43:44 selector.py:141] Using Flashinfer backend.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m WARNING 07-14 09:43:44 registry.py:205] `mm_limits` has already been set for model=/root/public-storage/models/Qwen/Qwen3.6-35B-A3B, and will be overwritten by the new values.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m INFO 07-14 09:43:44 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m INFO 07-14 09:43:44 selector.py:141] Using Flashinfer backend.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m WARNING 07-14 09:43:44 registry.py:205] `mm_limits` has already been set for model=/root/public-storage/models/Qwen/Qwen3.6-35B-A3B, and will be overwritten by the new values.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m INFO 07-14 09:43:44 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
|
||||||
|
INFO 07-14 09:43:45 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f56a01879d0>, local_subscribe_port=60197, remote_subscribe_port=None)
|
||||||
|
INFO 07-14 09:43:45 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m INFO 07-14 09:43:45 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m INFO 07-14 09:43:45 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m INFO 07-14 09:43:45 model_runner.py:1065] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m INFO 07-14 09:43:45 selector.py:141] Using Flashinfer backend.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m INFO 07-14 09:43:45 selector.py:141] Using Flashinfer backend.
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m INFO 07-14 09:43:45 selector.py:141] Using Flashinfer backend.
|
||||||
|
INFO 07-14 09:43:45 selector.py:141] Using Flashinfer backend.
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:45, 1.81s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:24, 1.02s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:04<00:32, 1.41s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:06<00:34, 1.59s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:06<00:28, 1.36s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:09<00:32, 1.60s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:09<00:23, 1.22s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:11<00:25, 1.42s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:12<00:21, 1.29s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:14<00:24, 1.52s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:14<00:16, 1.13s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:16<00:18, 1.35s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:18<00:19, 1.52s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:20<00:20, 1.71s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:21<00:15, 1.44s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:23<00:17, 1.72s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:24<00:12, 1.35s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:26<00:11, 1.50s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:27<00:10, 1.56s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:30<00:10, 1.82s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:31<00:07, 1.59s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:31<00:05, 1.34s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:34<00:04, 1.65s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:36<00:03, 1.75s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:38<00:01, 1.82s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.44s/it]
|
||||||
|
|
||||||
|
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.49s/it]
|
||||||
|
|
||||||
|
INFO 07-14 09:44:24 model_runner.py:1076] Loading model weights took 16.2303 GB
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m INFO 07-14 09:44:24 model_runner.py:1076] Loading model weights took 16.2303 GB
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m INFO 07-14 09:44:24 model_runner.py:1076] Loading model weights took 16.2303 GB
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m INFO 07-14 09:44:24 model_runner.py:1076] Loading model weights took 16.2303 GB
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks: 'NoneType' object is not callable, Traceback (most recent call last):
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker.py", line 223, in determine_num_available_blocks
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.model_runner.profile_run()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/multi_step_model_runner.py", line 661, in profile_run
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return self._base_model_runner.profile_run()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1314, in profile_run
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.execute_model(model_input, kv_caches, intermediate_tensors)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1641, in execute_model
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.attn_state.begin_forward(model_input)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/attention/backends/flashinfer.py", line 261, in begin_forward
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] model_input.attn_metadata.decode_wrapper = state._get_decode_wrapper()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/attention/backends/flashinfer.py", line 130, in _get_decode_wrapper
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self._decode_wrapper = BatchDecodeWithPagedKVCacheWrapper(
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] TypeError: 'NoneType' object is not callable
|
||||||
|
[1;36m(VllmWorkerProcess pid=6761)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231]
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks: 'NoneType' object is not callable, Traceback (most recent call last):
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker.py", line 223, in determine_num_available_blocks
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.model_runner.profile_run()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/multi_step_model_runner.py", line 661, in profile_run
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return self._base_model_runner.profile_run()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1314, in profile_run
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.execute_model(model_input, kv_caches, intermediate_tensors)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1641, in execute_model
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.attn_state.begin_forward(model_input)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/attention/backends/flashinfer.py", line 261, in begin_forward
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] model_input.attn_metadata.decode_wrapper = state._get_decode_wrapper()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/attention/backends/flashinfer.py", line 130, in _get_decode_wrapper
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self._decode_wrapper = BatchDecodeWithPagedKVCacheWrapper(
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] TypeError: 'NoneType' object is not callable
|
||||||
|
[1;36m(VllmWorkerProcess pid=6759)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231]
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks: 'NoneType' object is not callable, Traceback (most recent call last):
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] output = executor(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker.py", line 223, in determine_num_available_blocks
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.model_runner.profile_run()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/multi_step_model_runner.py", line 661, in profile_run
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return self._base_model_runner.profile_run()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1314, in profile_run
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.execute_model(model_input, kv_caches, intermediate_tensors)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] return func(*args, **kwargs)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/model_runner.py", line 1641, in execute_model
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self.attn_state.begin_forward(model_input)
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/attention/backends/flashinfer.py", line 261, in begin_forward
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] model_input.attn_metadata.decode_wrapper = state._get_decode_wrapper()
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] File "/usr/local/corex/lib/python3/dist-packages/vllm/attention/backends/flashinfer.py", line 130, in _get_decode_wrapper
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] self._decode_wrapper = BatchDecodeWithPagedKVCacheWrapper(
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231] TypeError: 'NoneType' object is not callable
|
||||||
|
[1;36m(VllmWorkerProcess pid=6760)[0;0m ERROR 07-14 09:44:24 multiproc_worker_utils.py:231]
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
|
||||||
|
return _run_code(code, main_globals, None,
|
||||||
|
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
|
||||||
|
exec(code, run_globals)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 595, in <module>
|
||||||
|
uvloop.run(run_server(args))
|
||||||
|
File "/usr/local/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
|
||||||
|
return loop.run_until_complete(wrapper())
|
||||||
|
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
|
||||||
|
File "/usr/local/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
|
||||||
|
return await main
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 562, in run_server
|
||||||
|
async with build_async_engine_client(args) as engine_client:
|
||||||
|
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
|
||||||
|
return await anext(self.gen)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 108, in build_async_engine_client
|
||||||
|
async with build_async_engine_client_from_engine_args(
|
||||||
|
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
|
||||||
|
return await anext(self.gen)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 142, in build_async_engine_client_from_engine_args
|
||||||
|
engine_client = await asyncio.get_running_loop().run_in_executor(
|
||||||
|
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
|
||||||
|
result = self.fn(*self.args, **self.kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 674, in from_engine_args
|
||||||
|
engine = cls(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 569, in __init__
|
||||||
|
self.engine = self._engine_class(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 265, in __init__
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/llm_engine.py", line 349, in __init__
|
||||||
|
self._initialize_kv_caches()
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/llm_engine.py", line 484, in _initialize_kv_caches
|
||||||
|
self.model_executor.determine_num_available_blocks())
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 39, in determine_num_available_blocks
|
||||||
|
num_blocks = self._run_workers("determine_num_available_blocks", )
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
|
||||||
|
import pynvml # type: ignore[import]
|
||||||
|
INFO 07-14 09:23:27 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
|
||||||
|
2026-07-14 09:23:29.177777: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
|
||||||
|
2026-07-14 09:23:29.228965: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
|
||||||
|
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
|
||||||
|
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
|
||||||
|
INFO 07-14 09:23:34 api_server.py:530] vLLM API server version 0.6.3
|
||||||
|
INFO 07-14 09:23:34 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=4096, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=4096, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=4, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
|
||||||
|
INFO 07-14 09:23:34 config.py:1670] Downcasting torch.float32 to torch.float16.
|
||||||
|
INFO 07-14 09:23:45 config.py:887] Defaulting to use mp for distributed inference
|
||||||
|
WARNING 07-14 09:23:45 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
|
||||||
|
INFO 07-14 09:23:45 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=4, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
|
||||||
|
INFO 07-14 09:23:46 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
|
||||||
|
INFO 07-14 09:23:46 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
|
||||||
|
INFO 07-14 09:23:46 selector.py:115] Using XFormers backend.
|
||||||
|
WARNING 07-14 09:23:46 registry.py:205] `mm_limits` has already been set for model=/root/public-storage/models/Qwen/Qwen3.6-35B-A3B, and will be overwritten by the new values.
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
|
||||||
|
return _run_code(code, main_globals, None,
|
||||||
|
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
|
||||||
|
exec(code, run_globals)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 595, in <module>
|
||||||
|
uvloop.run(run_server(args))
|
||||||
|
File "/usr/local/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
|
||||||
|
return loop.run_until_complete(wrapper())
|
||||||
|
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
|
||||||
|
File "/usr/local/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
|
||||||
|
return await main
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 562, in run_server
|
||||||
|
async with build_async_engine_client(args) as engine_client:
|
||||||
|
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
|
||||||
|
return await anext(self.gen)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 108, in build_async_engine_client
|
||||||
|
async with build_async_engine_client_from_engine_args(
|
||||||
|
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
|
||||||
|
return await anext(self.gen)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/entrypoints/openai/api_server.py", line 142, in build_async_engine_client_from_engine_args
|
||||||
|
engine_client = await asyncio.get_running_loop().run_in_executor(
|
||||||
|
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
|
||||||
|
result = self.fn(*self.args, **self.kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 674, in from_engine_args
|
||||||
|
engine = cls(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 569, in __init__
|
||||||
|
self.engine = self._engine_class(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/async_llm_engine.py", line 265, in __init__
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/engine/llm_engine.py", line 335, in __init__
|
||||||
|
self.model_executor = executor_class(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 215, in __init__
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/executor_base.py", line 47, in __init__
|
||||||
|
self._init_executor()
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 108, in _init_executor
|
||||||
|
self.driver_worker = self._create_worker(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/gpu_executor.py", line 105, in _create_worker
|
||||||
|
return create_worker(**self._get_create_worker_kwargs(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/executor/gpu_executor.py", line 24, in create_worker
|
||||||
|
wrapper.init_worker(**kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/worker_base.py", line 449, in init_worker
|
||||||
|
self.worker = worker_class(*args, **kwargs)
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/multi_step_worker.py", line 28, in __init__
|
||||||
|
self.model_runner = MultiStepModelRunner(
|
||||||
|
File "/usr/local/corex/lib/python3/dist-packages/vllm/worker/multi_step_model_runner.py", line 317, in __init__
|
||||||
|
raise ValueError(
|
||||||
|
ValueError: Multi-Step not supported for attention backend: xformers. Set VLLM_ATTENTION_BACKEND to a value from ['flash-attn', 'rocm-flash-attn', 'flashinfer'].
|
||||||
|
ERROR 07-14 09:23:46 multiproc_worker_utils.py:117] Worker VllmWorkerProcess pid 6686 died, exit code: -15
|
||||||
|
ERROR 07-14 09:23:46 multiproc_worker_utils.py:117] Worker VllmWorkerProcess pid 6687 died, exit code: -15
|
||||||
|
ERROR 07-14 09:23:46 multiproc_worker_utils.py:117] Worker VllmWorkerProcess pid 6688 died, exit code: -15
|
||||||
|
INFO 07-14 09:23:46 multiproc_worker_utils.py:121] Killing local vLLM worker processes
|
||||||
Reference in New Issue
Block a user