dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix(GDN): force fp16 cast before norm+out_proj — ixformer matmul requires kHalf

Browse Source 
matmul.cu:149 'Expected input.dtype() == kHalf' error in competition log.
Root cause: _torch_chunk_gated_delta_rule returns fp32 core_out,
passed directly to self.norm() → self.out_proj() which calls ixformer matmul.

Fix: explicit .to(torch.float16) on core_out and z before norm.
This commit is contained in:
project6-dev
2026-08-10 04:45:26 +00:00
parent d841c44e55
commit 7d4edd4ac7
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
qwen3_6_scripts/qwen3_5.py
View File

@@ -737,6 +737,9 @@ class GatedDeltaNet(nn.Module):
# Gate + norm + output proj
z = z_all[s:e].reshape(seq_len, local_num_v, self.head_v_dim)
core_out = core_out.reshape(seq_len, local_num_v, self.head_v_dim)
# Force fp16 — ixformer matmul requires kHalf
core_out = core_out.to(torch.float16)
z = z.to(torch.float16)
normed = self.norm(
core_out.reshape(-1, self.head_v_dim),
z.reshape(-1, self.head_v_dim))