EuroLLM-22B-Instruct-GPTQ/README.md

license, language, base_model, library_name, tags, pipeline_tag, model_creator, quantized_by

license	language	base_model	library_name	tags	pipeline_tag	model_creator	quantized_by
apache-2.0	en de fr it es pt pl nl tr sv cs el hu ro fi uk sk da bg no sl lt lv et hr ga mt	utter-project/EuroLLM-22B-Instruct-2512	transformers	gptq 4bit quantized eurollm multilingual european instruct	text-generation	UTTER Project	Euraika

EuroLLM-22B-Instruct GPTQ (4-bit)

This is a 4-bit GPTQ quantized version of utter-project/EuroLLM-22B-Instruct-2512.

Model Details

Attribute	Value
Original Model	EuroLLM-22B-Instruct-2512
Quantization	GPTQ 4-bit
Group Size	128
Activation Order	desc_act=True
Calibration Samples	512
Sequence Length	1024
Calibration Dataset	OpenHermes-2.5 (ChatML format)
Size	~14GB

Quantization Configuration

{
  "bits": 4,
  "group_size": 128,
  "desc_act": true,
  "damp_percent": 0.01,
  "sym": true,
  "true_sequential": true
}

Quantization Quality

The quantization achieved excellent loss values across all 53 transformer layers:

• Attention projections: 10⁻⁵ to 10⁻⁶ range
• MLP layers: 10⁻⁵ range
• No error explosion or instability

Languages Supported

EuroLLM supports all 24 official EU languages plus additional European languages: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Norwegian, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Turkish, Ukrainian

Usage

With GPTQModel (Recommended)

from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model = GPTQModel.load("Euraika/EuroLLM-22B-Instruct-GPTQ", device="cuda:0")
tokenizer = AutoTokenizer.from_pretrained("Euraika/EuroLLM-22B-Instruct-GPTQ")

prompt = """<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
Explain quantum computing in simple terms.<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With vLLM

from vllm import LLM, SamplingParams

llm = LLM(model="Euraika/EuroLLM-22B-Instruct-GPTQ", quantization="gptq")
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)

outputs = llm.generate(["Your prompt here"], sampling_params)

With Transformers + AutoGPTQ

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Euraika/EuroLLM-22B-Instruct-GPTQ",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Euraika/EuroLLM-22B-Instruct-GPTQ")

Chat Template

This model uses the ChatML format:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{assistant_message}<|im_end|>

Hardware Requirements

• Minimum: 16GB VRAM (RTX 4080, RTX A4000)
• Recommended: 24GB VRAM (RTX 4090, RTX A5000, RTX 6000)
• Supports multi-GPU inference via device_map="auto"

Credits

• Original Model: UTTER Project - EuroLLM team
• Quantization: Euraika using GPTQModel v5.6.12
• Calibration Dataset: OpenHermes-2.5

License

Apache 2.0 (same as base model)