---
library_name: transformers
license: apache-2.0
pretty_name: Qwen3-1.7B GPT-OSS-20B Distilled
base_model:
- Qwen/Qwen3-1.7B
tags:
- qwen
- distillation
- sft
- text-generation
- transformers
- peft

---
# Qwen3-1.7B-GPT-OSS-20B-Distilled

This model is a fine-tuned version of **`Qwen/Qwen3-1.7B`**, distilled from **`openai/gpt-oss-20b`** using LoRA and SFTTrainer.

## Model Description

This is a student model created through knowledge distillation from a larger, more capable teacher model. The goal is to transfer the knowledge and capabilities of the `openai/gpt-oss-20b` (teacher) to a smaller, more efficient `Qwen/Qwen3-1.7B` (student) model.


## Training Details
- **Teacher Model**: openai/gpt-oss-20b
- **Dataset**: Synthetic dataset of 1000 samples in Brazilian Portuguese generated by the teacher using random samples from [dominguesm/alpaca-data-pt-br](https://huggingface.co/datasets/dominguesm/alpaca-data-pt-br).
- **LoRA Config**: r=16, alpha=32, dropout=0.05
- **Training Hyperparams**: 3 epochs, learning rate=0.0002, batch size=2

## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(model, "wandgibaut/qwen-1.7b-gpt-oss-20b-distilled")
```