6cdf2ec87b
来源:
1. fla-org/flash-linear-attention (5538 stars)
→ upstream_ref/fla/ops/gated_delta_rule/naive.py (正确的纯 PyTorch GDN)
→ upstream_ref/fla/ops/gated_delta_rule/chunk.py (Triton chunk kernel)
→ upstream_ref/fla/layers/gated_deltanet.py (层集成)
2. vllm-project/vllm main (88717 stars)
→ upstream_ref/vllm_gdn/gdn/qwen_gdn_linear_attn.py (1751行, Qwen3.5 原生 GDN)
→ upstream_ref/vllm_gdn/ops/causal_conv1d.py (1289行, 正确的 Conv1d)
→ upstream_ref/vllm_gdn/third_party/ops/ (FLA Triton ops vendored)
→ upstream_ref/vllm_gdn/models/qwen3_5.py (vllm 最新 Qwen3.5 模型)
3. Deep-Spark/xllm (BI-V100 硬件厂商)
→ upstream_ref/xllm_latest/core/layers/npu_torch/qwen3_gated_delta_net_base.cpp (576行)
→ upstream_ref/xllm_latest/core/kernels/npu/npu_causal_conv1d.cpp
→ upstream_ref/xllm_latest/core/kernels/npu/npu_recurrent_gated_delta_rule.cpp
目的: 修复 corex_gdn.py Conv1d groups 接口不匹配问题
错误: conv1d_weight shape (2560,1,4) 被当成 (num_k_heads,1,4) 索引
conv_dim = key_dim*2 + value_dim = 10240, TP=4 后 2560
FLA naive.py 和 vllm qwen_gdn_linear_attn.py 有正确的实现可直接对接
43 lines
1.2 KiB
Python
43 lines
1.2 KiB
Python
# Copyright (c) 2023-2026, Songlin Yang, Yu Zhang, Zhiyuan Li
|
|
#
|
|
# This source code is licensed under the MIT license found in the
|
|
# LICENSE file in the root directory of this source tree.
|
|
# For a list of all contributors, visit:
|
|
# https://github.com/fla-org/flash-linear-attention/graphs/contributors
|
|
|
|
from fla.modules.conv import (
|
|
ImplicitLongConvolution,
|
|
LongConvolution,
|
|
PositionalEmbedding,
|
|
ShortConvolution,
|
|
causal_conv1d,
|
|
fft_conv,
|
|
)
|
|
from fla.modules.conv.cp import CausalConv1dFunctionCP, causal_conv1d_cp
|
|
from fla.modules.conv.cuda import FastCausalConv1dFn, fast_causal_conv1d_fn
|
|
from fla.modules.conv.triton import (
|
|
CausalConv1dFunction,
|
|
causal_conv1d_bwd,
|
|
causal_conv1d_fwd,
|
|
causal_conv1d_update,
|
|
causal_conv1d_update_states,
|
|
)
|
|
|
|
__all__ = [
|
|
'CausalConv1dFunction',
|
|
'CausalConv1dFunctionCP',
|
|
'FastCausalConv1dFn',
|
|
'ImplicitLongConvolution',
|
|
'LongConvolution',
|
|
'PositionalEmbedding',
|
|
'ShortConvolution',
|
|
'causal_conv1d',
|
|
'causal_conv1d_bwd',
|
|
'causal_conv1d_cp',
|
|
'causal_conv1d_fwd',
|
|
'causal_conv1d_update',
|
|
'causal_conv1d_update_states',
|
|
'fast_causal_conv1d_fn',
|
|
'fft_conv',
|
|
]
|