132 lines
4.5 KiB
Markdown
132 lines
4.5 KiB
Markdown
---
|
|
license: other
|
|
license_name: lfm1.0
|
|
license_link: https://huggingface.co/LiquidAI/LFM2.5-350M-Base/blob/main/LICENSE
|
|
base_model: LiquidAI/LFM2.5-350M-Base
|
|
tags:
|
|
- lfm2
|
|
- hssa
|
|
- hierarchical-sparse-attention
|
|
- sparse-attention
|
|
- long-context
|
|
- edge
|
|
- text-generation
|
|
- gguf
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
---
|
|
|
|
# LFM2.5-350M-HSSA
|
|
|
|
An experimental variant of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base)
|
|
in which the six full-attention (GQA) layers are replaced with **Hierarchical
|
|
Searchable Sparse Attention (HSSA)**, then continued-pretrained so the HSSA
|
|
routers learn content-dependent block selection on real data.
|
|
|
|
The convolutional (LIV) blocks of LFM2 are left untouched. Only the attention
|
|
layers `[2, 5, 8, 10, 12, 14]` are converted. The donor Q/K/V/O projections and
|
|
per-head Q/K norms are preserved; HSSA adds two small router projections
|
|
(`router_q_proj`, `router_k_proj`, dim 128) per converted layer.
|
|
|
|
- **Base model:** LiquidAI/LFM2.5-350M-Base
|
|
- **Parameters:** ~356M (+ ~1.6M router parameters)
|
|
- **Attention:** HSSA on the 6 attention layers — O(n log n) routed sparse
|
|
attention: local window (256) + top-8 routed blocks of 64 tokens, selected by
|
|
beam search over a balanced router tree
|
|
- **Precision:** bf16
|
|
- **Continued pretraining:** 2048-token windows, LR 2e-5, router supervision
|
|
loss weight 0.1
|
|
- **Intended use:** research on sparse attention / edge text generation
|
|
experiments
|
|
|
|
## GGUF builds (llama.cpp / Ollama / LM Studio)
|
|
|
|
Quantized GGUF builds are provided. **Important:** GGUF builds run with
|
|
**dense attention** — the HSSA router tensors are stripped during conversion,
|
|
because llama.cpp has no HSSA kernel. All fine-tuned backbone weights are
|
|
identical to this checkpoint, so outputs are close to (but not exactly) the
|
|
sparse-attention model.
|
|
|
|
| file | quant | size |
|
|
|---|---|---|
|
|
| `model-hssa-Q8_0.gguf` | Q8_0 | 379 MB |
|
|
| `model-hssa-Q6_K.gguf` | Q6_K | 293 MB |
|
|
| `model-hssa-Q4_K_M.gguf` | Q4_K_M | 229 MB |
|
|
|
|
```bash
|
|
llama-cli -m model-hssa-Q8_0.gguf -p "What is C. elegans?" -n 128
|
|
```
|
|
|
|
Q8_0 is recommended; at this model size Q4_K_M shows visible quality loss.
|
|
|
|
## Using with transformers
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "DarkKnighToS223/lfm2.5-350m-hssa"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")
|
|
```
|
|
|
|
Note: stock `transformers` loads the fine-tuned backbone with standard dense
|
|
attention; the HSSA router tensors in the checkpoint are ignored (same
|
|
behaviour as the GGUF builds).
|
|
|
|
The HSSA configuration is stored in `hssa_conversion.json`:
|
|
|
|
| knob | value |
|
|
|---|---|
|
|
| block_size | 64 |
|
|
| query_group_size | 16 |
|
|
| local_window | 256 |
|
|
| top_blocks | 8 |
|
|
| router_dim | 128 |
|
|
| router_beam_width | 16 |
|
|
|
|
## Evaluation
|
|
|
|
Synthetic long-context retrieval suite (RULER-style; needle-in-a-haystack,
|
|
multi-needle, associative recall, copy, variable tracing), 75 fixed prompts,
|
|
`answer_recall` scoring. Contexts of 512/1024 tokens are within the CPT
|
|
training window; 2048-token prompts fall outside it.
|
|
|
|
Continued pretraining lifts overall recall from 0.03 (freshly converted,
|
|
random routers) to 0.27.
|
|
|
|
| answer_recall | LFM2.5-350M-HSSA |
|
|
|---|---|
|
|
| overall (512+1024) | **0.27** |
|
|
| niah | **0.70** |
|
|
| multi_needle | **0.63** |
|
|
| associative_recall | 0.00 |
|
|
| copy | 0.00 |
|
|
| variable_trace | 0.00 |
|
|
|
|
Out-of-window contexts (2048-token targets, ~3.4k actual tokens): recall 0.08 —
|
|
degraded but non-zero, suggesting longer-window CPT would help. A dense
|
|
baseline of the same family scores higher on this suite; that gap is the
|
|
current cost of O(n log n) sparse attention after a short CPT run.
|
|
|
|
## Training context
|
|
|
|
Continued pretraining was short (a few thousand steps on 2048-token windows) —
|
|
enough for the HSSA routers to learn sensible block selection after
|
|
conversion. This is a research checkpoint, not a production-tuned model.
|
|
Expect quality below the original LFM2.5-350M on knowledge-heavy tasks; this
|
|
variant exists to study HSSA on a real, pretrained backbone.
|
|
|
|
## Limitations
|
|
|
|
- GGUF builds and stock `transformers` run with dense attention; the HSSA
|
|
routers are exercised only by the research training/eval framework.
|
|
- Trained on 2048-token windows; retrieval degrades beyond that range.
|
|
- Base-model knowledge cutoff and biases are inherited; factual accuracy is
|
|
limited at this scale.
|
|
|
|
## Acknowledgements
|
|
|
|
Built on LiquidAI's LFM2.5-350M-Base. HSSA conversion, continued pretraining
|