library_name, tags, license, datasets, base_model, pipeline_tag

library_name	tags	license	datasets	base_model	pipeline_tag
transformers	dpo alignment preference-optimization	apache-2.0	argilla/ultrafeedback-binarized-preferences-cleaned argilla/distilabel-capybara-dpo-7k-binarized argilla/distilabel-intel-orca-dpo-pairs	ayushshah/Qwen3-1.7B-UltraChat-SFT	text-generation

Qwen3-1.7B Chat

This model is a Direct Preference Optimized (DPO) variant of Qwen3-1.7B-UltraChat-SFT, which itself is a supervised fine-tuned version of Qwen3-1.7B-Base.

Starting from the SFT model, preference alignment was performed using Direct Preference Optimization (DPO) on multiple human preference datasets to improve response quality, reasoning, truthfulness, and conversational naturalness. The resulting model produces responses that are generally rich, more helpful, coherent, and better aligned with user preferences while preserving the conversational capabilities learned during supervised fine-tuning. This model does not support thinking, vision or tool calling yet.

Preference optimization was performed using LoRA adapters, which were merged into the base weights after training.

View the LoRA Adapter.
View the GitHub repository for the complete post-training pipeline.

Quantizations of this model are present in ayushshah/Qwen3-1.7B-Chat-GGUF. You can use them in LLaMA.cpp, Ollama or any other applications you like. This model and its quantizations are also available on Ollama. You can directly pull and use this model from Ollama. Visit the Ollama model page.

Model Overview

Qwen3-1.7B-Chat has the following features:

• Type: Causal Language Model
• Training Stage: Pretraining → Supervised Fine-Tuning → Direct Preference Optimization
• Number of Parameters: 1.7B
• Number of Parameters (Non-Embedding): 1.4B
• Number of Layers: 28
• Number of Attention Heads (GQA): 16 for Q and 8 for KV
• Context Length: 32,768

Training

Stage 1: Supervised Fine-Tuning

• Method: Supervised Fine-Tuning (SFT)
• Dataset: UltraChat-200k
• Framework: Unsloth
• Fine-Tuning: LoRA + embed_tokens + lm_head

Stage 2: Preference Alignment

• Method: Direct Preference Optimization (DPO)
• Frameworks: TRL + Unsloth
• Fine-Tuning: LoRA
• Precision: BF16
• Environment: Windows Subsystem for Linux (WSL)
• GPU: RTX 4090 24 GB using FlashAttention 2, xFormers and Triton

Dataset Preprocessing

1. The datasets were first filtered by removing rows with low chosen ratings. For argilla/distilabel-capybara-dpo-7k-binarized and argilla/ultrafeedback-binarized-preferences-cleaned, rows where the chosen rating was < 4 were removed. Similarly, for argilla/distilabel-intel-orca-dpo-pairs, rows where the preference was tie, the chosen rating was < 8, or in_gsm8k_train was true were removed.
2. For argilla/ultrafeedback-binarized-preferences-cleaned, only rows where the difference between the chosen and rejected ratings was >= 1 were retained.
3. Any row where either the chosen or rejected conversation exceeded 1024 tokens was removed.
4. The argilla/ultrafeedback-binarized-preferences-cleaned dataset was significantly larger than the other datasets; therefore, it was undersampled by randomly selecting 15,000 rows.
5. Finally, the datasets were concatenated and shuffled.

DPO Configuration

Parameter	Value
Epochs	1
Batch Size	2
Gradient Accumulation	16
Effective Batch Size	32
Gradient Checkpointing	True
Optimizer	AdamW 8-bit
Learning Rate	5e-6
LoRA Rank	32
LoRA Alpha	64
β	0.1
Scheduler	Cosine
Warmup	10%
Max seq length	1024

LoRA was applied to: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.

Evaluation

Evaluation was performed using EleutherAI's Language Model Evaluation Harness.

Benchmark	SFT	Chat	Δ
MMLU-Pro	37.91	39.86	+1.95
GSM8K	71.27	72.48	+1.21
HellaSwag	62.28	62.26	-0.02
TruthfulQA mc2	51.62	53.09	+1.47
ARC Challenge	44.97	43.26	-1.71
IFEval strict prompt	33.09	30.13	-2.96

Blind A/B Evaluation tests with LLM-as-a-judge and Human-In-The-Loop calibration preferred the DPO model over the SFT model in majority of the runs.

Usage

Unsloth (Recommended)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="ayushshah/Qwen3-1.7B-Chat",
    max_seq_length=2048,
    dtype="bfloat16",
)

FastLanguageModel.for_inference(model)

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "ayushshah/Qwen3-1.7B-Chat"
)

model = AutoModelForCausalLM.from_pretrained(
    "ayushshah/Qwen3-1.7B-Chat",
    torch_dtype="auto",
    device_map="auto"
)

Limitations

Although preference optimization improves conversational quality, reasoning, and helpfulness, this model is still limited by the capabilities and knowledge of the underlying Qwen3-1.7B architecture. It may occasionally generate incorrect or hallucinated information, struggle with complex multi-step reasoning, or fail to follow ambiguous instructions accurately. Furthermore, this model has not undergone reinforcement learning from human feedback (RLHF) or additional safety-specific alignment beyond DPO and should not be relied upon for high-stakes applications without appropriate human oversight.

Compared to the SFT model, the primary observed trade-off was a slight reduction in strict instruction-following ability, accompanied by a tendency to generate more elaborate responses and a slightly higher likelihood of hallucinations on complex queries.

Citation

@misc{qwen3technicalreport,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2025},
  eprint={2505.09388},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.09388}
}