44 lines
970 B
Markdown
44 lines
970 B
Markdown
---
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen3-4B
|
|
tags:
|
|
- qwen3
|
|
- lora
|
|
- fine-tuned
|
|
- alpaca
|
|
- chatml
|
|
- instruction-following
|
|
datasets:
|
|
- yahma/alpaca-cleaned
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# ChatWithMe — Qwen3-4B Fine-tuned on Alpaca
|
|
|
|
A Qwen3-4B model fine-tuned with LoRA on the Alpaca dataset for instruction-following conversations.
|
|
|
|
## Model Details
|
|
|
|
- **Base model:** Qwen/Qwen3-4B
|
|
- **Fine-tuning method:** LoRA (r=8, alpha=16)
|
|
- **Dataset:** yahma/alpaca-cleaned (~52k examples)
|
|
- **Chat format:** ChatML
|
|
- **Training:** 1 epoch, SFTTrainer, 4-bit quantization, bf16, cosine scheduler
|
|
- **Hardware:** NVIDIA A100 80GB
|
|
- **Final training loss:** 1.0875
|
|
- **Final validation loss:** 1.0976
|
|
- **Perplexity:** ~3.00
|
|
|
|
## LoRA Configuration
|
|
|
|
- **Target modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
|
- **Rank:** 8
|
|
- **Alpha:** 16
|
|
- **Dropout:** 0.05
|
|
- **Trainable parameters:** ~13M / 4B total (0.3%)
|
|
|
|
## Usage
|
|
|
|
```python |