Compare commits

...

2 Commits

Author SHA1 Message Date
Claude
1d9b620416 fix(crash): disable corex_gdn_chunk_recurrent — pybind signature mismatch
The .so's torch_chunk_gated_delta_rule() only accepts positional args:
  (Tensor, Tensor, Tensor, Tensor, Tensor, int, Optional[Tensor], bool, bool)
But Python calls it with keyword args:
  (q, k, v, g, beta, initial_state=, output_final_state=, use_qk_l2norm_in_kernel=)

This causes 'incompatible function arguments' crash during profiling
(determine_num_available_blocks), killing the engine before it starts.

Fix: _HAS_COREX_GDN_CHUNK = False, forcing Python _torch_chunk_gated_delta_rule.
This is what a3c45d3b effectively did (its .so wasn't compiled), explaining
why a3c45d3b works but aa4b4992 crashes.
2026-08-14 01:36:18 +00:00
project6-dev
716034bdd0 fix(OOM): lower blocks cap 5000→3000 — flash_attn needs ~4GB temp buffer
5000 blocks KV cache fills GPU memory, flash_attn_varlen_func OOMs
allocating temp attention buffer on first real request.
3000 blocks × 16 = 48K tokens capacity, leaves room for attention.
2026-08-14 01:33:06 +00:00
2 changed files with 10 additions and 5 deletions

View File

@@ -21,11 +21,12 @@ CAPACITY_REPLACEMENT = """\
num_gpu_blocks = reserve_block_major_gpu_blocks(
num_gpu_blocks, cache_block_size)
# BI100: profiling with zero-tensor attention underestimates memory.
# Hardcap at 5000 blocks (80K tokens) to prevent runtime OOM.
if num_gpu_blocks > 5000:
# Hardcap at 3000 blocks (48K tokens) to prevent runtime OOM.
# Must leave ~4GB free for flash_attn_varlen_func temp buffers.
if num_gpu_blocks > 3000:
logger.warning(
"[BI100] capping num_gpu_blocks: %d -> 5000", num_gpu_blocks)
num_gpu_blocks = 5000
"[BI100] capping num_gpu_blocks: %d -> 3000", num_gpu_blocks)
num_gpu_blocks = 3000
num_gpu_blocks = max(num_gpu_blocks, 0)
num_cpu_blocks = max(num_cpu_blocks, 0)
"""

View File

@@ -153,7 +153,11 @@ try:
except ImportError:
_corex_gdn_chunk_recurrent = None
_HAS_COREX_GDN_CHUNK = _corex_gdn_chunk_recurrent is not None
# DISABLED: corex_gdn_chunk_recurrent.so pybind uses positional args only,
# but the call site passes keyword args (initial_state=, output_final_state=).
# This causes "incompatible function arguments" crash during profiling.
# Force Python fallback until .so signature is fixed.
_HAS_COREX_GDN_CHUNK = False
from vllm.model_executor.models.interfaces import (HasInnerState, SupportsLoRA,
SupportsMultiModal)