EuroLLM-22B-Instruct-GPTQ/README.md

---
license: apache-2.0
language:
- en
- de
- fr
- it
- es
- pt
- pl
- nl
- tr
- sv
- cs
- el
- hu
- ro
- fi
- uk
- sk
- da
- bg
- no
- sl
- lt
- lv
- et
- hr
- ga
- mt
base_model: utter-project/EuroLLM-22B-Instruct-2512
library_name: transformers
tags:
- gptq
- 4bit
- quantized
- eurollm
- multilingual
- european
- instruct
pipeline_tag: text-generation
model_creator: UTTER Project
quantized_by: Euraika
---

# EuroLLM-22B-Instruct GPTQ (4-bit)

This is a 4-bit GPTQ quantized version of [utter-project/EuroLLM-22B-Instruct-2512](https://huggingface.co/utter-project/EuroLLM-22B-Instruct-2512).

## Model Details

| Attribute | Value |
|-----------|-------|
| Original Model | EuroLLM-22B-Instruct-2512 |
| Quantization | GPTQ 4-bit |
| Group Size | 128 |
| Activation Order | desc_act=True |
| Calibration Samples | 512 |
| Sequence Length | 1024 |
| Calibration Dataset | OpenHermes-2.5 (ChatML format) |
| Size | ~14GB |

## Quantization Configuration

```json
{
  "bits": 4,
  "group_size": 128,
  "desc_act": true,
  "damp_percent": 0.01,
  "sym": true,
  "true_sequential": true
}
```

## Quantization Quality

The quantization achieved excellent loss values across all 53 transformer layers:
- Attention projections: 10⁻⁵ to 10⁻⁶ range
- MLP layers: 10⁻⁵ range
- No error explosion or instability

## Languages Supported

EuroLLM supports all 24 official EU languages plus additional European languages:
Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Norwegian, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Turkish, Ukrainian

## Usage

### With GPTQModel (Recommended)

```python
from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model = GPTQModel.load("Euraika/EuroLLM-22B-Instruct-GPTQ", device="cuda:0")
tokenizer = AutoTokenizer.from_pretrained("Euraika/EuroLLM-22B-Instruct-GPTQ")

prompt = """<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
Explain quantum computing in simple terms.<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### With vLLM

```python
from vllm import LLM, SamplingParams

llm = LLM(model="Euraika/EuroLLM-22B-Instruct-GPTQ", quantization="gptq")
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)

outputs = llm.generate(["Your prompt here"], sampling_params)
```

### With Transformers + AutoGPTQ

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Euraika/EuroLLM-22B-Instruct-GPTQ",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Euraika/EuroLLM-22B-Instruct-GPTQ")
```

## Chat Template

This model uses the ChatML format:

```
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{assistant_message}<|im_end|>
```

## Hardware Requirements

- **Minimum**: 16GB VRAM (RTX 4080, RTX A4000)
- **Recommended**: 24GB VRAM (RTX 4090, RTX A5000, RTX 6000)
- Supports multi-GPU inference via device_map="auto"

## Credits

- **Original Model**: [UTTER Project](https://huggingface.co/utter-project) - EuroLLM team
- **Quantization**: Euraika using GPTQModel v5.6.12
- **Calibration Dataset**: [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)

## License

Apache 2.0 (same as base model)