40 lines
1.4 KiB
Markdown
40 lines
1.4 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
pretty_name: Qwen3-1.7B GPT-OSS-20B Distilled
|
|
base_model:
|
|
- Qwen/Qwen3-1.7B
|
|
tags:
|
|
- qwen
|
|
- distillation
|
|
- sft
|
|
- text-generation
|
|
- transformers
|
|
- peft
|
|
|
|
---
|
|
# Qwen3-1.7B-GPT-OSS-20B-Distilled
|
|
|
|
This model is a fine-tuned version of **`Qwen/Qwen3-1.7B`**, distilled from **`openai/gpt-oss-20b`** using LoRA and SFTTrainer.
|
|
|
|
## Model Description
|
|
|
|
This is a student model created through knowledge distillation from a larger, more capable teacher model. The goal is to transfer the knowledge and capabilities of the `openai/gpt-oss-20b` (teacher) to a smaller, more efficient `Qwen/Qwen3-1.7B` (student) model.
|
|
|
|
|
|
## Training Details
|
|
- **Teacher Model**: openai/gpt-oss-20b
|
|
- **Dataset**: Synthetic dataset of 1000 samples in Brazilian Portuguese generated by the teacher using random samples from [dominguesm/alpaca-data-pt-br](https://huggingface.co/datasets/dominguesm/alpaca-data-pt-br).
|
|
- **LoRA Config**: r=16, alpha=32, dropout=0.05
|
|
- **Training Hyperparams**: 3 epochs, learning rate=0.0002, batch size=2
|
|
|
|
## Usage
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
from peft import PeftModel
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", device_map="auto")
|
|
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
|
|
model = PeftModel.from_pretrained(model, "wandgibaut/qwen-1.7b-gpt-oss-20b-distilled")
|
|
```
|