初始化项目,由ModelHub XC社区提供模型
Model: Entrit/Qwen2.5-3B-trit-uniform-d3 Source: Original Platform
This commit is contained in:
60
README.md
Normal file
60
README.md
Normal file
@@ -0,0 +1,60 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: Qwen/Qwen2.5-3B
|
||||
tags:
|
||||
- quantization
|
||||
- ternary
|
||||
- balanced-ternary
|
||||
- tritllm
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# Qwen2.5-3B-trit-uniform-d3
|
||||
|
||||
Balanced ternary quantization of [`Qwen/Qwen2.5-3B`](https://huggingface.co/Qwen/Qwen2.5-3B) at depth **d=3** (27 levels per weight, **5.05 bits per weight**).
|
||||
|
||||
Produced with the codec from **"Balanced Ternary Post-Training Quantization for Large Language Models"** (Stentzel, 2026). See [Entrit/tritllm-codec](https://huggingface.co/Entrit/tritllm-codec) for the codec source.
|
||||
|
||||
## Quick load
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("Entrit/Qwen2.5-3B-trit-uniform-d3")
|
||||
tokenizer = AutoTokenizer.from_pretrained("Entrit/Qwen2.5-3B-trit-uniform-d3")
|
||||
```
|
||||
|
||||
The weights are dequantized to FP16 for stock-`transformers` compatibility. The on-disk size is therefore the same as the FP16 source. The 5.05-bpw figure refers to the *information content* of the quantized matrices and is what matters for inference on hardware that consumes the packed trit format directly (see [Entrit/tritllm-kernel](https://huggingface.co/Entrit/tritllm-kernel)).
|
||||
|
||||
## Quantization details
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Source model | [`Qwen/Qwen2.5-3B`](https://huggingface.co/Qwen/Qwen2.5-3B) |
|
||||
| Depth | d=3 (27 levels) |
|
||||
| Bits per weight | 5.05 |
|
||||
| Group size | 16 |
|
||||
| Scale codebook | 27-entry log-spaced (scale_depth=3) |
|
||||
| Method | Uniform PTQ |
|
||||
| Quantized layers | all 2D linear matrices |
|
||||
| Kept FP16 | `lm_head`, token embeddings, all `*_norm` layers |
|
||||
| Codec | tritllm v2 |
|
||||
|
||||
## Citation
|
||||
|
||||
```
|
||||
@article{stentzel2026ternaryptq,
|
||||
title = {Balanced Ternary Post-Training Quantization for Large Language Models},
|
||||
author = {Stentzel, Eric},
|
||||
year = 2026,
|
||||
note = {Entrit Systems}
|
||||
}
|
||||
```
|
||||
|
||||
## Reproducibility
|
||||
|
||||
```bash
|
||||
git clone https://huggingface.co/Entrit/tritllm-codec
|
||||
cd tritllm-codec
|
||||
python quantize_model_v2.py --model Qwen/Qwen2.5-3B --configs uniform-d3 --out ./out
|
||||
```
|
||||
Reference in New Issue
Block a user