fix(CRITICAL): max_model_len 100000→80000 (KV cache only 88112) + NaN fix
Docker log proves two fatal issues:
1. max_model_len=100000 > KV cache capacity 88112 → ValueError crash
'max seq len (100000) is larger than maximum number of tokens
that can be stored in KV cache (88112)'
Fix: set max_model_len=80000 (safe margin below 88112)
2. NaN in GatedDeltaNet layers 34,36,37,38 (frac=1.0000)
Root cause: g.cumsum() → g.exp() overflow to inf → inf*0 = NaN
Fix: clamp all g values to [-80,80] before exp() calls
(max safe float32 exp input ~88, use 80 for margin)
Applied to: cumsum result, k_cumdecay, attn_inter, last_state update
3. CoreX modules confirmed NOT in base image:
'CoreX GDN module not found'
'CoreX MoE module not found'
→ pure PyTorch is the only path, must be numerically stable
This commit is contained in:
@@ -8,7 +8,7 @@ command:
|
||||
- --served-model-name
|
||||
- llm
|
||||
- --max-model-len
|
||||
- '100000'
|
||||
- '80000'
|
||||
- --gpu-memory-utilization
|
||||
- '0.9'
|
||||
- --trust-remote-code
|
||||
|
||||
@@ -144,6 +144,9 @@ def _torch_chunk_gated_delta_rule(
|
||||
diagonal=0)
|
||||
|
||||
g = g.cumsum(dim=-1)
|
||||
# Numerical stability: clamp cumulative decay to prevent exp() → inf → NaN
|
||||
# CCCL pattern: block_scan overflow guard. Max safe float32 exp input ~88.
|
||||
g = g.clamp(-80.0, 80.0)
|
||||
decay_mask = ((g.unsqueeze(-1) - g.unsqueeze(-2)).tril().exp().float()).tril()
|
||||
attn = -((k_beta @ key.transpose(-1, -2)) * decay_mask).masked_fill(mask_upper, 0)
|
||||
for i in range(1, chunk_size):
|
||||
@@ -152,7 +155,7 @@ def _torch_chunk_gated_delta_rule(
|
||||
attn[..., i, :i] = row + (row.unsqueeze(-1) * sub).sum(-2)
|
||||
attn = attn + torch.eye(chunk_size, dtype=attn.dtype, device=attn.device)
|
||||
value = attn @ v_beta
|
||||
k_cumdecay = attn @ (k_beta * g.exp().unsqueeze(-1))
|
||||
k_cumdecay = attn @ (k_beta * g.clamp(-80, 80).exp().unsqueeze(-1))
|
||||
|
||||
last_state = (
|
||||
torch.zeros(batch, num_heads, k_dim, v_dim, dtype=value.dtype, device=value.device)
|
||||
@@ -169,11 +172,11 @@ def _torch_chunk_gated_delta_rule(
|
||||
attn_i = (q_i @ k_i.transpose(-1, -2) * decay_mask[:, :, i]).masked_fill_(mask_upper2, 0)
|
||||
v_prime = k_cumdecay[:, :, i] @ last_state
|
||||
v_new = v_i - v_prime
|
||||
attn_inter = (q_i * g[:, :, i, :, None].exp()) @ last_state
|
||||
attn_inter = (q_i * g[:, :, i, :, None].clamp(-80, 80).exp()) @ last_state
|
||||
core_out[:, :, i] = attn_inter + attn_i @ v_new
|
||||
last_state = (
|
||||
last_state * g[:, :, i, -1, None, None].exp()
|
||||
+ (k_i * (g[:, :, i, -1, None] - g[:, :, i]).exp()[..., None])
|
||||
last_state * g[:, :, i, -1, None, None].clamp(-80, 80).exp()
|
||||
+ (k_i * (g[:, :, i, -1, None] - g[:, :, i]).clamp(-80, 80).exp()[..., None])
|
||||
.transpose(-1, -2) @ v_new
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user