128 lines
5.1 KiB
Markdown
128 lines
5.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: flwrlabs/Lizzy-7B
|
||
tags:
|
||
- llama-cpp
|
||
- gguf
|
||
- olmo2
|
||
- quantized
|
||
- uk-english
|
||
- agentic
|
||
- function-calling
|
||
- lizzy-7B
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Lizzy-7B GGUF Quants
|
||
|
||
> 🚨 **Update:** Flower Labs has officially released their native GGUF quants.
|
||
> I highly recommend transitioning to their repository for the most stable inference and the corrected 32k context window: **[flwrlabs/Lizzy-7B-GGUF](https://huggingface.co/flwrlabs/Lizzy-7B-GGUF)**.
|
||
>
|
||
> *Note: During testing, I came across a bug with rope/context length issue, which has been patched in the official release. Thanks to the 250+ community members who tested this early build!*
|
||
|
||
**Quantized by [SolusOps](https://huggingface.co/SolusOps)**
|
||
|
||
**Original model:** [FlowerLabs/Lizzy-7B](https://huggingface.co/flwrlabs/Lizzy-7B)
|
||
|
||
**Official Quants:** [flwrlabs/Lizzy-7B-GGUF](https://huggingface.co/flwrlabs/Lizzy-7B-GGUF)
|
||
|
||
## About This Repo
|
||
|
||
This repository provides llama.cpp-compatible GGUF quants of **Lizzy-7B**, a UK-centric 7B language model built by [Flower Labs](https://flower.ai).
|
||
Refer to the [original model card](https://huggingface.co/flwrlabs/Lizzy-7B) for more details on the model.
|
||
|
||
## Available Quants
|
||
|
||
| File | Quant | Size | Use Case |
|
||
|---|---|---|---|
|
||
| `Lizzy-7B-f16.gguf` | F16 | ~14.6 GB | needs 20GB+ VRAM or CPU offload. |
|
||
| `Lizzy-7B-Q8_0.gguf` | Q8_0 | ~7.7 GB | **Recommended** fits 12GB VRAM with excellent context headroom. |
|
||
| `Lizzy-7B-Q6_K.gguf` | Q6_K | ~5.9 GB | for 10GB–12GB GPUs looking to maximize context size. |
|
||
| `Lizzy-7B-Q5_K_M.gguf` | Q5_K_M | ~5.1 GB | 8GB VRAM |
|
||
| `Lizzy-7B-Q4_K_M.gguf` | Q4_K_M | ~4.1 GB | 6GB–8GB GPUs. |
|
||
| `Lizzy-7B-Q3_K_M.gguf` | Q3_K_M | ~3.5 GB | edge devices, 4GB GPUs, or older laptops. |
|
||
|
||
## Hardware Tested
|
||
|
||
| Hardware | Quant | n_ctx | Speed |
|
||
|---|---|---|---|
|
||
| RTX 3060 12GB | Q8_0 | 8192 | ~23 tok/s |
|
||
| RTX 3060 12GB | F16 | 4096 | Slower (VRAM overflow to RAM) |
|
||
|
||
## Conversion Notes
|
||
|
||
### 1. Architecture: OLMo 2 Post-Norm Tensor Mapping
|
||
|
||
Lizzy-7B uses a Post-Norm variant of OLMo 2.
|
||
The standard convert_hf_to_gguf.py script does not recognise Flower Labs tensor naming conventions (post_attn_norm, post_mlp_norm)
|
||
and will fail or silently produce a broken file.
|
||
The fix was to register a LizzyForCausalLM model class in the llama.cpp conversion script,
|
||
subclassing Olmo2Model and overriding modify_tensors() to remap the four divergent tensor names:
|
||
|
||
```
|
||
python@ModelBase.register("LizzyForCausalLM")
|
||
class LizzyModel(Olmo2Model):
|
||
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
|
||
|
||
# 1. Lizzy: post_attn_norm -> llama.cpp: post_attention_norm
|
||
if name.endswith(".post_attn_norm.weight"):
|
||
yield (f"blk.{bid}.post_attention_norm.weight", data_torch)
|
||
return
|
||
|
||
# 2. Lizzy: post_mlp_norm -> llama.cpp: post_ffw_norm
|
||
if name.endswith(".post_mlp_norm.weight"):
|
||
yield (f"blk.{bid}.post_ffw_norm.weight", data_torch)
|
||
return
|
||
|
||
# 3. QK-Norms these mapped correctly via standard paths
|
||
if name.endswith(".q_norm.weight"):
|
||
yield (self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_Q_NORM, bid), data_torch)
|
||
return
|
||
|
||
if name.endswith(".k_norm.weight"):
|
||
yield (self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_K_NORM, bid), data_torch)
|
||
return
|
||
|
||
# 4. All other tensors — pass through normally
|
||
yield from super().modify_tensors(data_torch, name, bid)
|
||
|
||
```
|
||
|
||
No weights were altered. Only the tensor name metadata was remapped.
|
||
|
||
### 2. RoPE Scaling Factor Correction
|
||
|
||
During conversion, the script raised this warning:
|
||
|
||
```
|
||
The explicitly set RoPE scaling factor (config.rope_parameters['factor'] = 8.0)
|
||
does not match the ratio implicitly set by other parameters
|
||
(implicit factor = max_position_embeddings / original_max_position_embeddings = 4.0).
|
||
Using the explicit factor (8.0) in YaRN. This may cause unexpected behaviour.
|
||
```
|
||
|
||
The implicit factor (4.0) is mathematically derived from the model's own position embedding settings. The explicit `8.0` in the upstream config appears to be an authoring error. To produce a consistent and correctly-behaving GGUF, **the factor was corrected from `8.0` to `4.0`** in `config.json` before conversion.
|
||
|
||
This means the effective context window for these GGUFs reflects the 4.0× YaRN scaling, not 8.0×. If Flower Labs corrects the upstream config, a re-conversion would be straightforward.
|
||
|
||
## License
|
||
|
||
The original Lizzy-7B model is released under **Apache 2.0** by Flower Labs. These quants inherit that license.
|
||
|
||
## Links
|
||
|
||
- [Original Model: FlowerLabs/Lizzy-7B](https://huggingface.co/flwrlabs/Lizzy-7B)
|
||
- [Flower Labs](https://flower.ai)
|
||
- [llama.cpp](https://github.com/ggerganov/llama.cpp)
|
||
|
||
### About Me
|
||
|
||
This GGUF port was completed by **Anshuman Singh**.
|
||
|
||
* **GitHub:** [github.com/SolusOps](https://github.com/solusops)
|
||
* **LinkedIn:** [linkedin.com/in/anshumansingh2023](https://www.linkedin.com/in/anshumansingh2023/)
|
||
|
||
If this port helped your local deployment, feel free to connect!
|