fix(critical): DeltaNet NaN 99.98% — clamp gate logits before exp to prevent overflow
Docker log reveals: 'NaN in prefill GatedDeltaNet layer 0 (frac=0.9998)' Every DeltaNet (linear attention) layer produces 99.98% NaN values. nan_to_num replaces them with zeros, destroying model output quality. This is the root cause of d10_thinking_disable_ctk gibberish output. Root cause: g.cumsum(dim=-1) accumulates unbounded gate logits. When fed to exp(), large values overflow to Inf, which propagates as NaN through subsequent matmul and forward_sub operations. Fix: Clamp cumulative gate logits to [-20, 20] before any exp(). Range keeps exp in [~2e-9, ~5e8] — safe for float32 accumulation. Inspired by CCCL dispatch_reduce_deterministic.cuh: numerical stability requires bounded intermediate values (RFA pattern). Also in this log: - FusedMoE: 'vllm_moe_topk_softmax' not in ixformer → PyTorch fallback (expected, cannot fix without BI-V100 kernel rebuild) - OOM at end of sub168: 31.72 GiB GPU with 30.86 GiB allocated CCCL input: dispatch_reduce_deterministic.cuh RFA pattern, tuning_batch_memcpy.cuh (small=128t×4buf, large=256t×32B)
This commit is contained in:
@@ -112,6 +112,11 @@ def _torch_chunk_gated_delta_rule(
|
||||
diagonal=0)
|
||||
|
||||
g = g.cumsum(dim=-1)
|
||||
# Clamp gate logits to prevent exp overflow → NaN cascade.
|
||||
# CCCL dispatch_reduce_deterministic.cuh: numerical stability requires
|
||||
# bounded intermediate values. Gate logit range [-20, 20] keeps exp
|
||||
# in [~2e-9, ~5e8] — safe for float32 accumulation.
|
||||
g = g.clamp(-20.0, 20.0)
|
||||
decay_mask = ((g.unsqueeze(-1) - g.unsqueeze(-2)).tril().exp().float()).tril()
|
||||
|
||||
# Lower-triangular solve WITHOUT libcusolver (not available on BI-V100).
|
||||
|
||||
Reference in New Issue
Block a user