lfm2.5-350m-hssa/README.md

---
license: other
license_name: lfm1.0
license_link: https://huggingface.co/LiquidAI/LFM2.5-350M-Base/blob/main/LICENSE
base_model: LiquidAI/LFM2.5-350M-Base
tags:
  - lfm2
  - hssa
  - hierarchical-sparse-attention
  - sparse-attention
  - long-context
  - edge
  - text-generation
  - gguf
language:
  - en
pipeline_tag: text-generation
library_name: transformers
---

# LFM2.5-350M-HSSA

An experimental variant of [LiquidAI/LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base)
in which the six full-attention (GQA) layers are replaced with **Hierarchical
Searchable Sparse Attention (HSSA)**, then continued-pretrained so the HSSA
routers learn content-dependent block selection on real data.

The convolutional (LIV) blocks of LFM2 are left untouched. Only the attention
layers `[2, 5, 8, 10, 12, 14]` are converted. The donor Q/K/V/O projections and
per-head Q/K norms are preserved; HSSA adds two small router projections
(`router_q_proj`, `router_k_proj`, dim 128) per converted layer.

- **Base model:** LiquidAI/LFM2.5-350M-Base
- **Parameters:** ~356M (+ ~1.6M router parameters)
- **Attention:** HSSA on the 6 attention layers — O(n log n) routed sparse
  attention: local window (256) + top-8 routed blocks of 64 tokens, selected by
  beam search over a balanced router tree
- **Precision:** bf16
- **Continued pretraining:** 2048-token windows, LR 2e-5, router supervision
  loss weight 0.1
- **Intended use:** research on sparse attention / edge text generation
  experiments

## GGUF builds (llama.cpp / Ollama / LM Studio)

Quantized GGUF builds are provided. **Important:** GGUF builds run with
**dense attention** — the HSSA router tensors are stripped during conversion,
because llama.cpp has no HSSA kernel. All fine-tuned backbone weights are
identical to this checkpoint, so outputs are close to (but not exactly) the
sparse-attention model.

| file | quant | size |
|---|---|---|
| `model-hssa-Q8_0.gguf` | Q8_0 | 379 MB |
| `model-hssa-Q6_K.gguf` | Q6_K | 293 MB |
| `model-hssa-Q4_K_M.gguf` | Q4_K_M | 229 MB |

```bash
llama-cli -m model-hssa-Q8_0.gguf -p "What is C. elegans?" -n 128
```

Q8_0 is recommended; at this model size Q4_K_M shows visible quality loss.

## Using with transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "DarkKnighToS223/lfm2.5-350m-hssa"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")
```

Note: stock `transformers` loads the fine-tuned backbone with standard dense
attention; the HSSA router tensors in the checkpoint are ignored (same
behaviour as the GGUF builds).

The HSSA configuration is stored in `hssa_conversion.json`:

| knob | value |
|---|---|
| block_size | 64 |
| query_group_size | 16 |
| local_window | 256 |
| top_blocks | 8 |
| router_dim | 128 |
| router_beam_width | 16 |

## Evaluation

Synthetic long-context retrieval suite (RULER-style; needle-in-a-haystack,
multi-needle, associative recall, copy, variable tracing), 75 fixed prompts,
`answer_recall` scoring. Contexts of 512/1024 tokens are within the CPT
training window; 2048-token prompts fall outside it.

Continued pretraining lifts overall recall from 0.03 (freshly converted,
random routers) to 0.27.

| answer_recall | LFM2.5-350M-HSSA |
|---|---|
| overall (512+1024) | **0.27** |
| niah | **0.70** |
| multi_needle | **0.63** |
| associative_recall | 0.00 |
| copy | 0.00 |
| variable_trace | 0.00 |

Out-of-window contexts (2048-token targets, ~3.4k actual tokens): recall 0.08 —
degraded but non-zero, suggesting longer-window CPT would help. A dense
baseline of the same family scores higher on this suite; that gap is the
current cost of O(n log n) sparse attention after a short CPT run.

## Training context

Continued pretraining was short (a few thousand steps on 2048-token windows) —
enough for the HSSA routers to learn sensible block selection after
conversion. This is a research checkpoint, not a production-tuned model.
Expect quality below the original LFM2.5-350M on knowledge-heavy tasks; this
variant exists to study HSSA on a real, pretrained backbone.

## Limitations

- GGUF builds and stock `transformers` run with dense attention; the HSSA
  routers are exercised only by the research training/eval framework.
- Trained on 2048-token windows; retrieval degrades beyond that range.
- Base-model knowledge cutoff and biases are inherited; factual accuracy is
  limited at this scale.

## Acknowledgements

Built on LiquidAI's LFM2.5-350M-Base. HSSA conversion, continued pretraining