project6 c17c490e06 fix(GDN): remove pre-cumsum clamp — match xllm reference, fix 99.98% NaN
ROOT CAUSE: g.clamp(-5,2) before cumsum corrupted gate values.
The GDN algorithm computes decay_mask = exp(g_i - g_j) which is
numerically stable via subtraction cancelling cumsum growth.
Pre-clamping g distorts these differences → wrong decay rates → NaN.

xllm reference: qwen3_gated_delta_net_base.cpp lines 170-238
- cumsum first (no pre-clamp)
- difference form: (g_i_last - g[:, i]).exp() for state update
- k_cumdecay uses g.exp() directly (not clamped)

Removed: g.clamp(-5,2), g.clamp(-20,20), g_exp_cache, g_clamped
Added: xllm-style g_i_last/g_exp_term/k_g_exp state update
2026-08-10 07:44:03 +00:00
2026-08-07 08:43:51 +00:00
2026-07-30 17:03:23 +08:00
2026-08-07 07:47:01 +00:00

project_6

Description
No description provided
Readme 427 MiB
Languages
C++ 41.8%
Cuda 31.6%
Python 22.2%
C 2.1%
CMake 1.1%
Other 1.1%