---
library_name: transformers
tags:
- dpo
- alignment
- preference-optimization
license: apache-2.0
datasets:
- argilla/ultrafeedback-binarized-preferences-cleaned
- argilla/distilabel-capybara-dpo-7k-binarized
- argilla/distilabel-intel-orca-dpo-pairs
base_model:
- ayushshah/Qwen3-1.7B-UltraChat-SFT
pipeline_tag: text-generation
---

# Qwen3-1.7B Chat

This model is a **Direct Preference Optimized (DPO)** variant of [Qwen3-1.7B-UltraChat-SFT](https://huggingface.co/ayushshah/Qwen3-1.7B-UltraChat-SFT), which itself is a supervised fine-tuned version of [Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base).

Starting from the SFT model, preference alignment was performed using **Direct Preference Optimization (DPO)** on multiple human preference datasets to improve response quality, reasoning, truthfulness, and conversational naturalness. The resulting model produces responses that are generally rich, more helpful, coherent, and better aligned with user preferences while preserving the conversational capabilities learned during supervised fine-tuning. This model does not support thinking, vision or tool calling yet.

Preference optimization was performed using LoRA adapters, which were merged into the base weights after training.

View the [LoRA Adapter](https://huggingface.co/ayushshah/Qwen3-1.7B-Chat-LoRA).<br>
View the [GitHub](https://github.com/AyushShahh/Qwen3-1.7B-Post-Training/tree/main) repository for the complete post-training pipeline.

**Quantizations** of this model are present in [ayushshah/Qwen3-1.7B-Chat-GGUF](https://huggingface.co/ayushshah/Qwen3-1.7B-Chat-GGUF). You can use them in [LLaMA.cpp](https://llama-cpp.com/), [Ollama](https://ollama.com/) or any other applications you like. This model and its quantizations are also available on **Ollama**. You can directly pull and use this model from Ollama. Visit the Ollama [model page](https://ollama.com/ayushshah/qwen3-1.7b-chat).

## Model Overview

**Qwen3-1.7B-Chat** has the following features:

* **Type:** Causal Language Model
* **Training Stage:** Pretraining → Supervised Fine-Tuning → Direct Preference Optimization
* **Number of Parameters:** 1.7B
* **Number of Parameters (Non-Embedding):** 1.4B
* **Number of Layers:** 28
* **Number of Attention Heads (GQA):** 16 for Q and 8 for KV
* **Context Length:** 32,768

### Training

#### Stage 1: Supervised Fine-Tuning

* **Method:** Supervised Fine-Tuning (SFT)
* **Dataset:** UltraChat-200k
* **Framework:** Unsloth
* **Fine-Tuning:** LoRA + `embed_tokens` + `lm_head`

#### Stage 2: Preference Alignment

* **Method:** Direct Preference Optimization (DPO)
* **Frameworks:** TRL + Unsloth
* **Fine-Tuning:** LoRA
* **Precision:** BF16
* **Environment:** Windows Subsystem for Linux (WSL)
* **GPU:** RTX 4090 24 GB using FlashAttention 2, xFormers and Triton

### Dataset Preprocessing

1. The datasets were first filtered by removing rows with low chosen ratings. For `argilla/distilabel-capybara-dpo-7k-binarized` and `argilla/ultrafeedback-binarized-preferences-cleaned`, rows where the chosen rating was `< 4` were removed. Similarly, for `argilla/distilabel-intel-orca-dpo-pairs`, rows where the preference was `tie`, the chosen rating was `< 8`, or `in_gsm8k_train` was `true` were removed.
2. For `argilla/ultrafeedback-binarized-preferences-cleaned`, only rows where the difference between the chosen and rejected ratings was `>= 1` were retained.
3. Any row where either the chosen or rejected conversation exceeded 1024 tokens was removed.
4. The `argilla/ultrafeedback-binarized-preferences-cleaned` dataset was significantly larger than the other datasets; therefore, it was undersampled by randomly selecting `15,000` rows.
5. Finally, the datasets were concatenated and shuffled.


### DPO Configuration

| Parameter             | Value        |
| --------------------- | ------------ |
| Epochs                | 1            |
| Batch Size            | 2            |
| Gradient Accumulation | 16           |
| Effective Batch Size  | 32           |
| Gradient Checkpointing | True        |
| Optimizer             | AdamW 8-bit  |
| Learning Rate         | 5e-6 |
| LoRA Rank             | 32           |
| LoRA Alpha            | 64           |
| β                     | 0.1  |
| Scheduler             | Cosine       |
| Warmup                | 10%          |
| Max seq length        | 1024         |

LoRA was applied to: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`.

## Evaluation

Evaluation was performed using EleutherAI's [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness/).

| Benchmark                       | SFT   | Chat  | Δ         |
| ------------------------------- | ----- | ----- | --------- |
| MMLU-Pro                        | 37.91 | 39.86 | **+1.95** |
| GSM8K                           | 71.27 | 72.48 | **+1.21** |
| HellaSwag                       | 62.28 | 62.26 | -0.02     |
| TruthfulQA <sub>mc2</sub>       | 51.62 | 53.09 | **+1.47** |
| ARC Challenge                   | 44.97 | 43.26 | -1.71     |
| IFEval <sub>strict prompt</sub> | 33.09 | 30.13 | -2.96     |

Blind **A/B Evaluation** tests with **LLM-as-a-judge** and **Human-In-The-Loop** calibration preferred the DPO model over the SFT model in majority of the runs.

## Usage

### Unsloth (Recommended)

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="ayushshah/Qwen3-1.7B-Chat",
    max_seq_length=2048,
    dtype="bfloat16",
)

FastLanguageModel.for_inference(model)
```

### transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "ayushshah/Qwen3-1.7B-Chat"
)

model = AutoModelForCausalLM.from_pretrained(
    "ayushshah/Qwen3-1.7B-Chat",
    torch_dtype="auto",
    device_map="auto"
)
```

## Limitations

Although preference optimization improves conversational quality, reasoning, and helpfulness, this model is still limited by the capabilities and knowledge of the underlying Qwen3-1.7B architecture. It may occasionally generate incorrect or hallucinated information, struggle with complex multi-step reasoning, or fail to follow ambiguous instructions accurately. Furthermore, this model has not undergone reinforcement learning from human feedback (RLHF) or additional safety-specific alignment beyond DPO and should not be relied upon for high-stakes applications without appropriate human oversight.

Compared to the SFT model, the primary observed trade-off was a slight reduction in strict instruction-following ability, accompanied by a tendency to generate more elaborate responses and a slightly higher likelihood of hallucinations on complex queries.

## Citation

```bibtex
@misc{qwen3technicalreport,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2025},
  eprint={2505.09388},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.09388}
}
```