Docker log reveals: 'NaN in prefill GatedDeltaNet layer 0 (frac=0.9998)'
Every DeltaNet (linear attention) layer produces 99.98% NaN values.
nan_to_num replaces them with zeros, destroying model output quality.
This is the root cause of d10_thinking_disable_ctk gibberish output.
Root cause: g.cumsum(dim=-1) accumulates unbounded gate logits.
When fed to exp(), large values overflow to Inf, which propagates
as NaN through subsequent matmul and forward_sub operations.
Fix: Clamp cumulative gate logits to [-20, 20] before any exp().
Range keeps exp in [~2e-9, ~5e8] — safe for float32 accumulation.
Inspired by CCCL dispatch_reduce_deterministic.cuh: numerical
stability requires bounded intermediate values (RFA pattern).
Also in this log:
- FusedMoE: 'vllm_moe_topk_softmax' not in ixformer → PyTorch fallback
(expected, cannot fix without BI-V100 kernel rebuild)
- OOM at end of sub168: 31.72 GiB GPU with 30.86 GiB allocated
CCCL input: dispatch_reduce_deterministic.cuh RFA pattern,
tuning_batch_memcpy.cuh (small=128t×4buf, large=256t×32B)
BI-V100 base image does not have libcusolver.so at:
/opt/sw_home/local/cuda/lib64/libcusolver.so
torch.linalg.solve_triangular requires cuSOLVER which is missing.
Replace with row-by-row forward substitution using only basic
matmul and indexing ops (torch.zeros_like, matmul, indexing).
The linear_attention gated_delta_rule solves (I-A)@X=RHS where A
is strictly lower-triangular. Forward sub: x[0]=rhs[0],
x[i]=rhs[i]+A[i,:i]@x[:i]. Mathematically equivalent.
CCCL source read: cub/device/dispatch/dispatch_reduce_by_key.cuh
- DeviceReduceByKey sorts input by key, pads to tile boundary, then
one fused kernel processes all key-value segments in parallel.
- This is architecturally identical to base engine's fused_moe.py:
moe_align_block_size (sort+pad) → invoke_fused_moe_kernel (one launch).
Discovery: _custom_ops.py (line 776-806) confirms ixformer HAS native MoE:
- ixf_F.vllm_moe_topk_softmax
- ixf_F.vllm_moe_align_block_size
- ixf_F.vllm_invoke_fused_moe_kernel (takes only BLOCK_SIZE_M config)
Previous code assumed 'ixformer lacks MoE kernels' and used _pure_pytorch_experts
(Python for-loop over 256 experts). This may have been wrong or outdated.
Change: MoeSparseBlock.forward now tries self.experts (FusedMoE native) first.
If the native kernel fails on BI-V100, it catches the exception, logs a warning,
and permanently falls back to _pure_pytorch_experts for that instance.
Impact if native works: one fused CUDA kernel vs 256× F.linear calls = massive
decode speedup. Impact if native fails: same behavior as before (fallback).
Qwen3.6-35B-A3B has 256 experts × top_k=8. The baseline prefill MoE:
for eid in unique_eids: # up to 256 iterations
tokens = hidden_states[tok_ids] # SCATTERED gather
F.linear(tokens, w13[eid])
Problem: hidden_states[tok_ids] creates a non-contiguous gather for each expert.
With 16384 tokens × 256 experts, this is 256 scattered gathers per layer.
Optimization (CCCL segmented-sort pattern):
1. Flatten all token-expert pairs: (T×K,) assignments
2. Sort by expert ID: tokens for same expert become CONTIGUOUS
3. Each F.linear gets contiguous input → much better memory access
4. Activation (silu × up) computed in ONE fused op across all pairs
5. index_add_ scatter-back is one kernel call
Memory access improvement:
Before: 256 × hidden_states[random_indices] → scattered HBM reads
After: sorted_tokens[start:end] → sequential HBM reads per expert
The expert loop still exists (can't batch variable-size GEMMs with F.linear),
but each iteration reads contiguous memory instead of scattered indices.
