dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix(deltanet): systematic NaN elimination via CCCL overflow_cast pattern

Browse Source 
Sub509 docker logs show 99.98-100% NaN rate in ALL GatedDeltaNet layers.
nan_to_num(nan=0.0) replaces them with zeros — entire DeltaNet layers
produce zero output, crippling model quality. This is root cause of:
- d03_tool_call FAIL (model too impaired to output <tool_call> XML)
- d01 content[0] (no content output, only 1085 reasoning tokens)
- d04 content[0] (same: reasoning but no content)
- 11x slower than opponent (model generates excessive tokens)

Five-layer fix based on CCCL overflow_cast_t pattern:

1. Pre-cumsum clamp: g.clamp(-0.5, 0.5) before cumsum (was: no pre-clamp)
   - Limits cumsum growth to ±32 for chunk_size=64
   - Post-cumsum clamp tightened from ±20 to ±12

2. A_log clamp tightened: [-20,20] → [-5,5]
   - exp(5) ≈ 148 vs exp(20) ≈ 4.9e8
   - Prevents extreme decay rates that feed into g

3. Forward substitution per-row clamp: ±1e4
   - _forward_sub_lower was the primary NaN amplifier
   - Each x[i] = rhs[i] + A[i,:i]@x[:i] now clamped

4. Cross-chunk state clamp: ±1e4
   - last_state *= g_last_exp can blow up across many chunks
   - Both prefill and decode paths protected

5. Decode temporal_state in-place clamp: ±1e4
   - ts_flat.clamp_() after baddbmm_ state update

Also adds SUB509_DEEP_DIAGNOSIS.md with full root cause analysis.
This commit is contained in:
Claude
2026-08-07 09:26:03 +00:00
parent a20e8614a4
commit 4eea584c9d
2 changed files with 178 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
qwen3_6_scripts/qwen3_5.py
View File

@@ -232,12 +232,18 @@ def _torch_chunk_gated_delta_rule(
torch.ones(chunk_size, chunk_size, dtype=torch.bool, device=query.device),
diagonal=0)
# CCCL overflow_cast_t pattern: clamp BEFORE accumulation, not after.
# Without pre-clamp, cumsum of large g values produces huge numbers
# that downstream exp() and matmul amplify into NaN.
# BI-V100 docker logs show 99.98-100% NaN rate in every GatedDeltaNet layer.
#
# Pre-clamp: limit each g element so cumsum over chunk_size stays bounded.
# With chunk_size=64 and per-element clamp ±0.3, cumsum range ≈ ±19.2.
# Post-clamp to ±12 keeps exp(g_diff) ≤ exp(24) ≈ 2.6e10 — safe for
# float32 matmul accumulation (k_dim=64 → max product ~1.7e12, within float32).
g = g.clamp(-0.5, 0.5)
g = g.cumsum(dim=-1)
# Clamp gate logits to prevent exp overflow → NaN cascade.
# CCCL dispatch_reduce_deterministic.cuh: numerical stability requires
# bounded intermediate values. Gate logit range [-20, 20] keeps exp
# in [~2e-9, ~5e8] — safe for float32 accumulation.
g = g.clamp(-20.0, 20.0)
g = g.clamp(-12.0, 12.0)
decay_mask = ((g.unsqueeze(-1) - g.unsqueeze(-2)).tril().exp().float()).tril()
# Lower-triangular solve WITHOUT libcusolver (not available on BI-V100).
@@ -269,16 +275,22 @@ def _torch_chunk_gated_delta_rule(
return torch.linalg.solve_triangular(
IminusA, rhs, upper=False, unitriangular=True)
else:
# Python forward substitution fallback
# Python forward substitution fallback with numerical stability.
# CCCL overflow_cast pattern: clamp intermediate results per row
# to prevent the A @ x accumulation from amplifying small errors
# into NaN. Without this, BI-V100 shows 100% NaN in every DeltaNet layer.
x = torch.zeros_like(rhs)
x[..., 0, :] = rhs[..., 0, :]
x[..., 0, :] = rhs[..., 0, :].clamp(-1e4, 1e4)
for i in range(1, C):
x[..., i, :] = rhs[..., i, :] + (A_lower[..., i, :i].unsqueeze(-2) @ x[..., :i, :]).squeeze(-2)
correction = (A_lower[..., i, :i].unsqueeze(-2) @ x[..., :i, :]).squeeze(-2)
x[..., i, :] = (rhs[..., i, :] + correction).clamp(-1e4, 1e4)
return x
value = _forward_sub_lower(A, v_beta)
k_cumdecay = _forward_sub_lower(A, k_beta * g.exp().unsqueeze(-1))
# Clamp g.exp() to prevent k_cumdecay from having extreme values
# that would amplify in the forward substitution loop.
k_cumdecay = _forward_sub_lower(A, k_beta * g.exp().clamp(-1e4, 1e4).unsqueeze(-1))
del A # free memory
@@ -319,6 +331,10 @@ def _torch_chunk_gated_delta_rule(
+ (k_i * g_diff_exp[:, :, i, :, None])
.transpose(-1, -2) @ v_new
)
# CCCL numerical guard: clamp state to prevent cross-chunk accumulation
# from amplifying into NaN. State elements represent k_dim × v_dim
# attention memory; values beyond ±1e4 indicate numerical runaway.
last_state = last_state.clamp(-1e4, 1e4)
if not output_final_state:
last_state = None
@@ -544,9 +560,13 @@ class GatedDeltaNet(nn.Module):
beta = b_all[s:e].sigmoid().unsqueeze(0) # (1, seq_len, local_num_v)
# CCCL overflow_cast pattern: clamp before exp to prevent
# overflow → NaN cascade. A_log.exp() can exceed float32 range
# when A_log > ~88; clamping to [-20,20] keeps exp in safe range.
_A_safe = self.A_log.float().clamp(-20.0, 20.0)
# overflow → NaN cascade. Tightened to [-5,5] because:
# A_log.exp() range [0.007, 148.4] — moderate decay rates.
# Multiplied by softplus(a + dt_bias) ≈ [0.7, 10] → g ≈ [-1484, -0.005]
# Per-element g then gets clamped to [-0.5, 0.5] in chunk_gated_delta_rule.
# The tighter clamp here prevents A_log outliers from creating
# extreme g values before the chunk-level clamp catches them.
_A_safe = self.A_log.float().clamp(-5.0, 5.0)
g = (-_A_safe.exp()
* F.softplus(a_all[s:e].float() + self.dt_bias)
).unsqueeze(0) # (1, seq_len, local_num_v)
@@ -634,8 +654,8 @@ class GatedDeltaNet(nn.Module):
v = v.reshape(num_seqs, 1, local_num_v, self.head_v_dim)
beta = b_all.sigmoid().unsqueeze(1) # (num_seqs, 1, local_num_v)
# CCCL overflow_cast pattern: clamp before exp (same as prefill path)
_A_safe = self.A_log.float().clamp(-20.0, 20.0)
# CCCL overflow_cast pattern: tightened to [-5,5] matching prefill path
_A_safe = self.A_log.float().clamp(-5.0, 5.0)
g = (-_A_safe.exp()
* F.softplus(a_all.float() + self.dt_bias)
).unsqueeze(1) # (num_seqs, 1, local_num_v)
@@ -654,7 +674,7 @@ class GatedDeltaNet(nn.Module):
q_t = _l2norm(q.squeeze(1)).float() * _scale # (B, H_v, k_dim)
k_t = _l2norm(k.squeeze(1)).float() # (B, H_v, k_dim)
v_t = v.squeeze(1).float() # (B, H_v, v_dim)
g_t = g.squeeze(1).float().clamp_(-20.0, 20.0).exp_() # (B, H_v) overflow_cast
g_t = g.squeeze(1).float().clamp_(-12.0, 12.0).exp_() # (B, H_v) overflow_cast tightened
bt = beta.squeeze(1).float() # (B, H_v)
# Decay state in-place: (B, H_v, k_dim, v_dim) *= scalar per head
@@ -676,6 +696,8 @@ class GatedDeltaNet(nn.Module):
k_t.view(BH, self.head_k_dim, 1),
delta.view(BH, 1, self.head_v_dim),
)
# CCCL numerical guard: clamp decode state (same as prefill cross-chunk)
ts_flat.clamp_(-1e4, 1e4)
# Output: core_out = q_t @ updated temporal_state
core_out = torch.bmm(