初始化项目，由ModelHub XC社区提供模型

Model: DarkKnighToS223/lfm2.5-350m-hssa Source: Original Platform
2026-07-27 22:29:52 +08:00
commit 7766eab7c2
11 changed files with 324160 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,41 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model-hssa-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+model-hssa-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+model-hssa-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+lfm2.5-350m-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+lfm2.5-350m-hssa-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+lfm2.5-350m-hssa-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,131 @@
+---
+license: other
+license_name: lfm1.0
+license_link: https://huggingface.co/LiquidAI/LFM2.5-350M-Base/blob/main/LICENSE
+base_model: LiquidAI/LFM2.5-350M-Base
+tags:
+  - lfm2
+  - hssa
+  - hierarchical-sparse-attention
+  - sparse-attention
+  - long-context
+  - edge
+  - text-generation
+  - gguf
+language:
+  - en
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# LFM2.5-350M-HSSA
+
+An experimental variant of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base)
+in which the six full-attention (GQA) layers are replaced with **Hierarchical
+Searchable Sparse Attention (HSSA)**, then continued-pretrained so the HSSA
+routers learn content-dependent block selection on real data.
+
+The convolutional (LIV) blocks of LFM2 are left untouched. Only the attention
+layers `[2, 5, 8, 10, 12, 14]` are converted. The donor Q/K/V/O projections and
+per-head Q/K norms are preserved; HSSA adds two small router projections
+(`router_q_proj`, `router_k_proj`, dim 128) per converted layer.
+
+- **Base model:** LiquidAI/LFM2.5-350M-Base
+- **Parameters:** ~356M (+ ~1.6M router parameters)
+- **Attention:** HSSA on the 6 attention layers — O(n log n) routed sparse
+  attention: local window (256) + top-8 routed blocks of 64 tokens, selected by
+  beam search over a balanced router tree
+- **Precision:** bf16
+- **Continued pretraining:** 2048-token windows, LR 2e-5, router supervision
+  loss weight 0.1
+- **Intended use:** research on sparse attention / edge text generation
+  experiments
+
+## GGUF builds (llama.cpp / Ollama / LM Studio)
+
+Quantized GGUF builds are provided. **Important:** GGUF builds run with
+**dense attention** — the HSSA router tensors are stripped during conversion,
+because llama.cpp has no HSSA kernel. All fine-tuned backbone weights are
+identical to this checkpoint, so outputs are close to (but not exactly) the
+sparse-attention model.
+
+| file | quant | size |
+|---|---|---|
+| `model-hssa-Q8_0.gguf` | Q8_0 | 379 MB |
+| `model-hssa-Q6_K.gguf` | Q6_K | 293 MB |
+| `model-hssa-Q4_K_M.gguf` | Q4_K_M | 229 MB |
+
+```bash
+llama-cli -m model-hssa-Q8_0.gguf -p "What is C. elegans?" -n 128
+```
+
+Q8_0 is recommended; at this model size Q4_K_M shows visible quality loss.
+
+## Using with transformers
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "DarkKnighToS223/lfm2.5-350m-hssa"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")
+```
+
+Note: stock `transformers` loads the fine-tuned backbone with standard dense
+attention; the HSSA router tensors in the checkpoint are ignored (same
+behaviour as the GGUF builds).
+
+The HSSA configuration is stored in `hssa_conversion.json`:
+
+| knob | value |
+|---|---|
+| block_size | 64 |
+| query_group_size | 16 |
+| local_window | 256 |
+| top_blocks | 8 |
+| router_dim | 128 |
+| router_beam_width | 16 |
+
+## Evaluation
+
+Synthetic long-context retrieval suite (RULER-style; needle-in-a-haystack,
+multi-needle, associative recall, copy, variable tracing), 75 fixed prompts,
+`answer_recall` scoring. Contexts of 512/1024 tokens are within the CPT
+training window; 2048-token prompts fall outside it.
+
+Continued pretraining lifts overall recall from 0.03 (freshly converted,
+random routers) to 0.27.
+
+| answer_recall | LFM2.5-350M-HSSA |
+|---|---|
+| overall (512+1024) | **0.27** |
+| niah | **0.70** |
+| multi_needle | **0.63** |
+| associative_recall | 0.00 |
+| copy | 0.00 |
+| variable_trace | 0.00 |
+
+Out-of-window contexts (2048-token targets, ~3.4k actual tokens): recall 0.08 —
+degraded but non-zero, suggesting longer-window CPT would help. A dense
+baseline of the same family scores higher on this suite; that gap is the
+current cost of O(n log n) sparse attention after a short CPT run.
+
+## Training context
+
+Continued pretraining was short (a few thousand steps on 2048-token windows) —
+enough for the HSSA routers to learn sensible block selection after
+conversion. This is a research checkpoint, not a production-tuned model.
+Expect quality below the original LFM2.5-350M on knowledge-heavy tasks; this
+variant exists to study HSSA on a real, pretrained backbone.
+
+## Limitations
+
+- GGUF builds and stock `transformers` run with dense attention; the HSSA
+  routers are exercised only by the research training/eval framework.
+- Trained on 2048-token windows; retrieval degrades beyond that range.
+- Base-model knowledge cutoff and biases are inherited; factual accuracy is
+  limited at this scale.
+
+## Acknowledgements
+
+Built on LiquidAI's LFM2.5-350M-Base. HSSA conversion, continued pretraining
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,7 @@
+{{- bos_token -}}{%- set system_prompt = "" -%}{%- set ns = namespace(system_prompt="") -%}{%- if messages[0]["role"] == "system" -%} {%- set ns.system_prompt = messages[0]["content"] -%} {%- set messages = messages[1:] -%}{%- endif -%}{%- if tools -%} {%- set ns.system_prompt = ns.system_prompt + ("
+" if ns.system_prompt else "") + "List of tools: <|tool_list_start|>[" -%} {%- for tool in tools -%} {%- if tool is not string -%} {%- set tool = tool | tojson -%} {%- endif -%} {%- set ns.system_prompt = ns.system_prompt + tool -%} {%- if not loop.last -%} {%- set ns.system_prompt = ns.system_prompt + ", " -%} {%- endif -%} {%- endfor -%} {%- set ns.system_prompt = ns.system_prompt + "]<|tool_list_end|>" -%}{%- endif -%}{%- if ns.system_prompt -%} {{- "<|im_start|>system
+" + ns.system_prompt + "<|im_end|>
+" -}}{%- endif -%}{%- for message in messages -%} {{- "<|im_start|>" + message["role"] + "
+" -}} {%- set content = message["content"] -%} {%- if content is not string -%} {%- set content = content | tojson -%} {%- endif -%} {%- if message["role"] == "tool" -%} {%- set content = "<|tool_response_start|>" + content + "<|tool_response_end|>" -%} {%- endif -%} {{- content + "<|im_end|>
+" -}}{%- endfor -%}{%- if add_generation_prompt -%} {{- "<|im_start|>assistant
+" -}}{%- endif -%}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,96 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "full_attn_idxs": null,
+  "hidden_size": 1024,
+  "hssa_conversion": {
+    "adapter_class": "LfmHSSAAttentionAdapter",
+    "converted_layers": [
+      2,
+      5,
+      8,
+      10,
+      12,
+      14
+    ],
+    "format_version": 1,
+    "hssa_config": {
+      "attention_backend": "torch",
+      "attention_dropout": 0.0,
+      "block_size": 64,
+      "head_dim": 64,
+      "hidden_size": 1024,
+      "local_window": 256,
+      "num_attention_heads": 16,
+      "num_key_value_heads": 8,
+      "prefill_group_chunk_size": 32,
+      "qk_norm": false,
+      "query_group_size": 16,
+      "router_aux_max_groups": 256,
+      "router_aux_num_candidates": 32,
+      "router_aux_target_temperature": 0.25,
+      "router_beam_width": 16,
+      "router_dim": 128,
+      "router_score_bias": 0.25,
+      "router_temperature": 1.0,
+      "top_blocks": 8
+    },
+    "model_type": "lfm2",
+    "source_model": "models/LFM2.5-350M-Base",
+    "transformers_version": "5.13.1"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.13.1",
+  "use_cache": true,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 7,
+  "pad_token_id": 0,
+  "transformers_version": "5.13.1"
+}
--- a/hssa_conversion.json
+++ b/hssa_conversion.json
@@ -0,0 +1,36 @@
+{
+  "adapter_class": "LfmHSSAAttentionAdapter",
+  "converted_layers": [
+    2,
+    5,
+    8,
+    10,
+    12,
+    14
+  ],
+  "format_version": 1,
+  "hssa_config": {
+    "attention_backend": "torch",
+    "attention_dropout": 0.0,
+    "block_size": 64,
+    "head_dim": 64,
+    "hidden_size": 1024,
+    "local_window": 256,
+    "num_attention_heads": 16,
+    "num_key_value_heads": 8,
+    "prefill_group_chunk_size": 32,
+    "qk_norm": false,
+    "query_group_size": 16,
+    "router_aux_max_groups": 256,
+    "router_aux_num_candidates": 32,
+    "router_aux_target_temperature": 0.25,
+    "router_beam_width": 16,
+    "router_dim": 128,
+    "router_score_bias": 0.25,
+    "router_temperature": 1.0,
+    "top_blocks": 8
+  },
+  "model_type": "lfm2",
+  "source_model": "models/LFM2.5-350M-Base",
+  "transformers_version": "5.13.1"
+}
--- a/lfm2.5-350m-Q4_K_M.gguf
+++ b/lfm2.5-350m-Q4_K_M.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5f7c6b9a7dd5b07130e75c26b80d46388537b9341575a4d2120942bb3bf6bc7
+size 229310336
--- a/lfm2.5-350m-hssa-Q6_K.gguf
+++ b/lfm2.5-350m-hssa-Q6_K.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3257fcb2e49eb8b01da49cfe7b4277d8d36fd1526b47bc65734cfeb143ce8b9
+size 293379968
--- a/lfm2.5-350m-hssa-Q8_0.gguf
+++ b/lfm2.5-350m-hssa-Q8_0.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e4f1291ba4c10f5b1b040e413b3d77cd4f3095386c40e3ad1d4b0902e8a2558
+size 379215744
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,21 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "is_local": true,
+  "legacy": false,
+  "local_files_only": true,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}