5a3bcbc247
CCCL overflow_cast.h pattern applied to qwen3_5.py: - Prefill gate: A_log.float().clamp(-20,20).exp() prevents NaN cascade - Decode gate: same clamp before exp (was unprotected, unlike prefill path) - Decode g_t: clamp_(-20,20) before in-place exp_() (was raw exp_()) Docker logs show 99.98% NaN in GatedDeltaNet layers — these unprotected exp() calls are the root cause. CCCL checked_allocator.cuh pattern applied to model_runner.py: - Wrap model forward in try/except torch.cuda.OutOfMemoryError - On OOM: empty_cache + gc.collect + retry once - Competitor Sub168 died permanently at layernorm x.float() OOM during replay (docker log evidence). This recovery keeps server alive. Source: cccl_upstream/libcudacxx/include/cuda/__numeric/overflow_cast.h Source: cccl_upstream/c2h/include/c2h/checked_allocator.cuh