---
language:
- fr
- en
- ar
base_model:
- Qwen/Qwen2.5-7B-Instruct
tags:
  - gguf
  - quantization
  - qwen2.5
  - llama.cpp
  - internal-audit
  - question-answering
  - fine-tuned
---

# Qwen2.5 – Internal Audit Q&A (Quantized GGUF)

This repository contains quantized GGUF-format variants of a **fine-tuned Qwen 2.5 model**, specialized for **question answering (Q&A) on internal audit data**.

These models are optimized for **efficient deployment** in environments using `llama.cpp`, `llama-cpp-python`, or compatible inference servers (e.g., `llama-server`, `text-generation-webui`).

## Fine-Tuning Overview

- **Base Model**: [Qwen2.5 7B](https://huggingface.co/Qwen/Qwen2.5-7B-instruct)
- **Fine-Tuning Task**: Instruction-based Q&A on internal audit reports, policies, and compliance logs
- **Training Data**: ~100k entries from anonymized internal audit datasets (private & proprietary)
- **Format**: Chat-style instruction tuning with questions and detailed answers

---

## 🗃️ Quantized Variants

| Filename              | Quantization | Description                                              |
|-----------------------|--------------|----------------------------------------------------------|
| `model-Q3_K_M.gguf`   | Q3_K_M       | 3-bit quantization — low memory footprint                |
| `model-Q4_K_M.gguf`   | Q4_K_M       | 4-bit — good performance and efficiency                  |
| `model-Q5_K_M.gguf`   | Q5_K_M       | 5-bit — balance between performance and quality          |
| `model-Q6_K.gguf`     | Q6_K         | 6-bit — high quality, higher RAM usage                   |
| `model-Q8_0.gguf`     | Q8_0         | 8-bit — near original model fidelity                     |
| `model-fp16.gguf`     | FP16         | Full precision — highest quality, requires GPU          |

---

##  ChatML Format

### Token structure

Each message in the conversation is wrapped like this:
```
<|im_start|>{role}
{message content}
<|im_end|>
```

- `{role}` is usually `system`, `user`, or `assistant`
- This clearly defines message boundaries for the model to interpret dialogue turns