Sub509 docker logs show 99.98-100% NaN rate in ALL GatedDeltaNet layers.
nan_to_num(nan=0.0) replaces them with zeros — entire DeltaNet layers
produce zero output, crippling model quality. This is root cause of:
- d03_tool_call FAIL (model too impaired to output <tool_call> XML)
- d01 content[0] (no content output, only 1085 reasoning tokens)
- d04 content[0] (same: reasoning but no content)
- 11x slower than opponent (model generates excessive tokens)
Five-layer fix based on CCCL overflow_cast_t pattern:
1. Pre-cumsum clamp: g.clamp(-0.5, 0.5) before cumsum (was: no pre-clamp)
- Limits cumsum growth to ±32 for chunk_size=64
- Post-cumsum clamp tightened from ±20 to ±12
2. A_log clamp tightened: [-20,20] → [-5,5]
- exp(5) ≈ 148 vs exp(20) ≈ 4.9e8
- Prevents extreme decay rates that feed into g
3. Forward substitution per-row clamp: ±1e4
- _forward_sub_lower was the primary NaN amplifier
- Each x[i] = rhs[i] + A[i,:i]@x[:i] now clamped
4. Cross-chunk state clamp: ±1e4
- last_state *= g_last_exp can blow up across many chunks
- Both prefill and decode paths protected
5. Decode temporal_state in-place clamp: ±1e4
- ts_flat.clamp_() after baddbmm_ state update
Also adds SUB509_DEEP_DIAGNOSIS.md with full root cause analysis.
4eea584c9d