72 lines
2.8 KiB
Markdown
72 lines
2.8 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen2.5-0.5B-Instruct
|
|
tags:
|
|
- text-generation-inference
|
|
- transformers
|
|
- qwen
|
|
- knowledge-distillation
|
|
- hidden-state-distillation
|
|
language:
|
|
- en
|
|
metrics:
|
|
- perplexity
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# Qwen2.5-0.5B-HiddenDistilled
|
|
|
|
This repository contains the **fully merged base + adapter weights** for Qwen2.5-0.5B-Instruct distilled from the teacher model **Qwen2.5-3B-Instruct**. The distillation pipeline optimizes a composite objective combining supervised learning cross-entropy, logit-level KL divergence, and MSE alignment of projected hidden states.
|
|
|
|
* **Base Student Model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
|
* **Teacher Model:** [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct)
|
|
* **LoRA Adapter Repo:** [sarimahsan101/Qwen2.5-0.5B-HiddenDistilled-LoRA](https://huggingface.co/sarimahsan101/Qwen2.5-0.5B-HiddenDistilled-LoRA)
|
|
* **Training Code (GitHub):** [sarimahsan101/distillation-hiddenstates](https://github.com/sarimahsan/distillation-hiddenstates-code)
|
|
|
|
|
|
## 📊 Trial Run Evaluation Metrics
|
|
|
|
Evaluation metrics compiled on a single **NVIDIA Tesla T4 (16GB)** GPU for **1 epoch** on a subset of Dolly, Alpaca SFT, and Ultrachat:
|
|
|
|
| Metric | Before Distillation | After Distillation | Change | Status |
|
|
| :--- | :---: | :---: | :---: | :---: |
|
|
| **Validation Perplexity** | 5.0924 | 5.2620 | +0.1696 | ✗ |
|
|
| **Teacher-Student KL Divergence** | 2.7913 | 1.9637 | -0.8276 | ✓ |
|
|
| **Hidden State Cosine Similarity** | 0.0075 | 0.0054 | -0.0021 | ✗ |
|
|
|
|
## 🚀 How to Use (Merged Model)
|
|
|
|
You can load this model directly using standard Hugging Face Transformers:
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_name = "sarimahsan101/Qwen2.5-0.5B-HiddenDistilled"
|
|
|
|
# Load tokenizer and merged model
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_name,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True
|
|
)
|
|
model.eval()
|
|
|
|
# Inference example
|
|
messages = [{"role": "user", "content": "Explain gravity in one sentence."}]
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(**inputs, max_new_tokens=50)
|
|
|
|
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## 🛠️ Training Configurations & Details
|
|
* **Framework:** PyTorch & Hugging Face Transformers / Trainer
|
|
* **Quantization:** 4-bit NF4 double quantization (`bitsandbytes`) for training, adapter weights were merged with FP16 base student.
|
|
* **Loss Weights:** Cross-Entropy: 0.3, KL Divergence: 0.4, Hidden MSE: 0.3
|