162 lines
3.5 KiB
Markdown
162 lines
3.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
- de
|
|
- fr
|
|
- it
|
|
- es
|
|
- pt
|
|
- pl
|
|
- nl
|
|
- tr
|
|
- sv
|
|
- cs
|
|
- el
|
|
- hu
|
|
- ro
|
|
- fi
|
|
- uk
|
|
- sk
|
|
- da
|
|
- bg
|
|
- no
|
|
- sl
|
|
- lt
|
|
- lv
|
|
- et
|
|
- hr
|
|
- ga
|
|
- mt
|
|
base_model: utter-project/EuroLLM-22B-Instruct-2512
|
|
library_name: transformers
|
|
tags:
|
|
- gptq
|
|
- 4bit
|
|
- quantized
|
|
- eurollm
|
|
- multilingual
|
|
- european
|
|
- instruct
|
|
pipeline_tag: text-generation
|
|
model_creator: UTTER Project
|
|
quantized_by: Euraika
|
|
---
|
|
|
|
# EuroLLM-22B-Instruct GPTQ (4-bit)
|
|
|
|
This is a 4-bit GPTQ quantized version of [utter-project/EuroLLM-22B-Instruct-2512](https://huggingface.co/utter-project/EuroLLM-22B-Instruct-2512).
|
|
|
|
## Model Details
|
|
|
|
| Attribute | Value |
|
|
|-----------|-------|
|
|
| Original Model | EuroLLM-22B-Instruct-2512 |
|
|
| Quantization | GPTQ 4-bit |
|
|
| Group Size | 128 |
|
|
| Activation Order | desc_act=True |
|
|
| Calibration Samples | 512 |
|
|
| Sequence Length | 1024 |
|
|
| Calibration Dataset | OpenHermes-2.5 (ChatML format) |
|
|
| Size | ~14GB |
|
|
|
|
## Quantization Configuration
|
|
|
|
```json
|
|
{
|
|
"bits": 4,
|
|
"group_size": 128,
|
|
"desc_act": true,
|
|
"damp_percent": 0.01,
|
|
"sym": true,
|
|
"true_sequential": true
|
|
}
|
|
```
|
|
|
|
## Quantization Quality
|
|
|
|
The quantization achieved excellent loss values across all 53 transformer layers:
|
|
- Attention projections: 10⁻⁵ to 10⁻⁶ range
|
|
- MLP layers: 10⁻⁵ range
|
|
- No error explosion or instability
|
|
|
|
## Languages Supported
|
|
|
|
EuroLLM supports all 24 official EU languages plus additional European languages:
|
|
Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Norwegian, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Turkish, Ukrainian
|
|
|
|
## Usage
|
|
|
|
### With GPTQModel (Recommended)
|
|
|
|
```python
|
|
from gptqmodel import GPTQModel
|
|
from transformers import AutoTokenizer
|
|
|
|
model = GPTQModel.load("Euraika/EuroLLM-22B-Instruct-GPTQ", device="cuda:0")
|
|
tokenizer = AutoTokenizer.from_pretrained("Euraika/EuroLLM-22B-Instruct-GPTQ")
|
|
|
|
prompt = """<|im_start|>system
|
|
You are a helpful AI assistant.<|im_end|>
|
|
<|im_start|>user
|
|
Explain quantum computing in simple terms.<|im_end|>
|
|
<|im_start|>assistant
|
|
"""
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
|
|
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
### With vLLM
|
|
|
|
```python
|
|
from vllm import LLM, SamplingParams
|
|
|
|
llm = LLM(model="Euraika/EuroLLM-22B-Instruct-GPTQ", quantization="gptq")
|
|
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)
|
|
|
|
outputs = llm.generate(["Your prompt here"], sampling_params)
|
|
```
|
|
|
|
### With Transformers + AutoGPTQ
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"Euraika/EuroLLM-22B-Instruct-GPTQ",
|
|
device_map="auto"
|
|
)
|
|
tokenizer = AutoTokenizer.from_pretrained("Euraika/EuroLLM-22B-Instruct-GPTQ")
|
|
```
|
|
|
|
## Chat Template
|
|
|
|
This model uses the ChatML format:
|
|
|
|
```
|
|
<|im_start|>system
|
|
{system_message}<|im_end|>
|
|
<|im_start|>user
|
|
{user_message}<|im_end|>
|
|
<|im_start|>assistant
|
|
{assistant_message}<|im_end|>
|
|
```
|
|
|
|
## Hardware Requirements
|
|
|
|
- **Minimum**: 16GB VRAM (RTX 4080, RTX A4000)
|
|
- **Recommended**: 24GB VRAM (RTX 4090, RTX A5000, RTX 6000)
|
|
- Supports multi-GPU inference via device_map="auto"
|
|
|
|
## Credits
|
|
|
|
- **Original Model**: [UTTER Project](https://huggingface.co/utter-project) - EuroLLM team
|
|
- **Quantization**: Euraika using GPTQModel v5.6.12
|
|
- **Calibration Dataset**: [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
|
|
|
|
## License
|
|
|
|
Apache 2.0 (same as base model)
|