71 lines
3.3 KiB
Markdown
71 lines
3.3 KiB
Markdown
---
|
||
license: llama3.1
|
||
base_model: meta-llama/Llama-3.1-8B
|
||
tags:
|
||
- flashnorm
|
||
- transformer-tricks
|
||
- efficient-inference
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Llama-3.1-8B-FlashNorm
|
||
|
||
FlashNorm-prepared compatibility checkpoint of [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B), derived from Meta's original weights obtained via the [unsloth/Llama-3.1-8B](https://huggingface.co/unsloth/Llama-3.1-8B) ungated mirror (bit-identical to the upstream release).
|
||
|
||
The FlashNorm transformation is mathematically exact. This checkpoint loads in stock `transformers` and `vLLM` without any code changes.
|
||
|
||
## What is FlashNorm?
|
||
|
||
An exact reformulation of `RMSNorm → Linear` that (i) folds the per-channel normalization weights into the following linear layer (`W* = W · diag(g)`) and (ii) defers the scalar `1/RMS(x)` normalization to after the matmul. On hardware with distinct vector and matrix units, the matrix multiplication and the RMS reduction can execute in parallel.
|
||
|
||
See the [paper](https://github.com/OpenMachine-ai/transformer-tricks/blob/main/doc/flashNorm.pdf) and the [transformer-tricks](https://github.com/OpenMachine-ai/transformer-tricks) repo for details.
|
||
|
||
## What’s different from the source checkpoint?
|
||
|
||
| Tensor | Source | This checkpoint |
|
||
|---|---|---|
|
||
| `model.layers.*.input_layernorm.weight` | learned per-channel `g` | all ones |
|
||
| `model.layers.*.self_attn.{q,k,v}_proj.weight` | `W` | `W · diag(g_input_layernorm)` |
|
||
| `model.layers.*.post_attention_layernorm.weight` | learned per-channel `g` | all ones |
|
||
| `model.layers.*.mlp.{gate,up}_proj.weight` | `W` | `W · diag(g_post_attention_layernorm)` |
|
||
| `model.norm.weight` | learned per-channel `g` | all ones |
|
||
| `lm_head.weight` | `W` | `W · diag(g_model_norm)` |
|
||
|
||
Llama-3.1-8B has untied embeddings, so `model.norm` is also folded into `lm_head`. All tensors are stored in the source dtype (`bfloat16`); merged products are computed in float32 internally before casting back.
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
tok = AutoTokenizer.from_pretrained('open-machine/Llama-3.1-8B-FlashNorm')
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
'open-machine/Llama-3.1-8B-FlashNorm',
|
||
dtype=torch.float16,
|
||
).cuda().eval()
|
||
|
||
ids = tok('Once upon a time there was', return_tensors='pt').input_ids.cuda()
|
||
out = model.generate(ids, max_new_tokens=50, do_sample=False)
|
||
print(tok.decode(out[0], skip_special_tokens=True))
|
||
```
|
||
|
||
With vLLM:
|
||
|
||
```bash
|
||
vllm serve open-machine/Llama-3.1-8B-FlashNorm
|
||
```
|
||
|
||
## Framework behavior
|
||
|
||
The FlashNorm transformation is mathematically exact.
|
||
|
||
- **HuggingFace Transformers at fp32**: greedy generation is bit-identical to the source.
|
||
- **HuggingFace Transformers at fp16** and **vLLM** (any precision): a one-token argmax flip is possible at tight decision points; downstream greedy decoding then amplifies this. Reason: precomputed merged weights interact differently with lossy inference kernels than runtime `x·g·W` would.
|
||
|
||
This is a general property of precomputing weight-folded tensors for lossy-inference kernels, **not specific to FlashNorm**. A native fused `RMSNorm + QKV` kernel (deferring `g` to runtime) eliminates the framework dependency and is in progress for vLLM / FlashInfer.
|
||
|
||
## License
|
||
|
||
Llama 3.1 Community License, inherited from the source model.
|