Selene-1-Mini-Llama-3.1-8B-GPTQ-W4A16/README.md

base_model, library_name, language, pipeline_tag, tags, license

base_model	library_name	language	pipeline_tag	tags	license
AtlaAI/Selene-1-Mini-Llama-3.1-8B	transformers	en de fr it pt es	text-generation	llama atla evaluation llm-as-a-judge meta conversational lm-judge llama-cpp gptq	llama3.1

🛝 Playground | 📄 Technical report | 💻 GitHub | 👀 Atla agent evals

<style> @keyframes rainbow { 0% { background-position: 0% 50%; } 50% { background-position: 100% 50%; } 100% { background-position: 0% 50%; } } </style> <style> @keyframes rainbow { 0% { background-position: 0% 50%; } 50% { background-position: 100% 50%; } 100% { background-position: 0% 50%; } } </style>

AtlaAI/Selene-1-Mini-Llama-3.1-8B-GPTQ-W4A16

This model was quantised into a 4-bit (W4A16) format using GPTQ from AtlaAI/Selene-1-Mini-Llama-3.1-8B. This was done using vLLM's llm-compressor library (https://docs.vllm.ai/en/latest/features/quantization/int4.html)

Refer to the original model card for more details on the model.

This quantisation was calibrated using a sample of 512 datapoints from the data used to train Selene-1-Mini. As a result, our quantised models show minimal performance degradation, losing <0.5% overall across benchmarks!

For reference, a GPTQ quantized 8-bit Llama-3.1-8B shows ~1.5% degradation across benchmarks.