dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

Compare commits

base: dylanyunlong:c8a982c4e8c9f2fb59957ad1f4eaa430b113517b
Branches Tags
dylanyunlong:main
...
compare: dylanyunlong:1d9b620416a29bd700bac058682dea3c43c373a5
Branches Tags
dylanyunlong:main

2 Commits
c8a982c4e8 ... 1d9b620416

Author SHA1 Message Date
Claude
1d9b620416 fix(crash): disable corex_gdn_chunk_recurrent — pybind signature mismatch 
The .so's torch_chunk_gated_delta_rule() only accepts positional args:
  (Tensor, Tensor, Tensor, Tensor, Tensor, int, Optional[Tensor], bool, bool)
But Python calls it with keyword args:
  (q, k, v, g, beta, initial_state=, output_final_state=, use_qk_l2norm_in_kernel=)

This causes 'incompatible function arguments' crash during profiling
(determine_num_available_blocks), killing the engine before it starts.

Fix: _HAS_COREX_GDN_CHUNK = False, forcing Python _torch_chunk_gated_delta_rule.
This is what a3c45d3b effectively did (its .so wasn't compiled), explaining
why a3c45d3b works but aa4b4992 crashes.
 2026-08-14 01:36:18 +00:00
project6-dev
716034bdd0 fix(OOM): lower blocks cap 5000→3000 — flash_attn needs ~4GB temp buffer 
5000 blocks KV cache fills GPU memory, flash_attn_varlen_func OOMs
allocating temp attention buffer on first real request.
3000 blocks × 16 = 48K tokens capacity, leaves room for attention.
 2026-08-14 01:33:06 +00:00
2 changed files with 10 additions and 5 deletions
Expand all files Collapse all files

9
qwen3_6_scripts/patch_block_major_worker_capacity.py
View File

@@ -21,11 +21,12 @@ CAPACITY_REPLACEMENT = """\
num_gpu_blocks = reserve_block_major_gpu_blocks( num_gpu_blocks = reserve_block_major_gpu_blocks(
num_gpu_blocks, cache_block_size) num_gpu_blocks, cache_block_size)
# BI100: profiling with zero-tensor attention underestimates memory. # BI100: profiling with zero-tensor attention underestimates memory.
# Hardcap at 5000 blocks (80K tokens) to prevent runtime OOM. # Hardcap at 3000 blocks (48K tokens) to prevent runtime OOM.
if num_gpu_blocks > 5000: # Must leave ~4GB free for flash_attn_varlen_func temp buffers.
if num_gpu_blocks > 3000:
logger.warning( logger.warning(
"[BI100] capping num_gpu_blocks: %d -> 5000", num_gpu_blocks) "[BI100] capping num_gpu_blocks: %d -> 3000", num_gpu_blocks)
num_gpu_blocks = 5000 num_gpu_blocks = 3000
num_gpu_blocks = max(num_gpu_blocks, 0) num_gpu_blocks = max(num_gpu_blocks, 0)
num_cpu_blocks = max(num_cpu_blocks, 0) num_cpu_blocks = max(num_cpu_blocks, 0)
""" """

6
qwen3_6_scripts/qwen3_5.py
View File

@@ -153,7 +153,11 @@ try:
except ImportError: except ImportError:
_corex_gdn_chunk_recurrent = None _corex_gdn_chunk_recurrent = None
_HAS_COREX_GDN_CHUNK = _corex_gdn_chunk_recurrent is not None # DISABLED: corex_gdn_chunk_recurrent.so pybind uses positional args only,
# but the call site passes keyword args (initial_state=, output_final_state=).
# This causes "incompatible function arguments" crash during profiling.
# Force Python fallback until .so signature is fixed.
_HAS_COREX_GDN_CHUNK = False
from vllm.model_executor.models.interfaces import (HasInnerState, SupportsLoRA, from vllm.model_executor.models.interfaces import (HasInnerState, SupportsLoRA,
SupportsMultiModal) SupportsMultiModal)