116 lines
3.6 KiB
Markdown
116 lines
3.6 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- qwen2.5
|
|
- sft
|
|
- openorca
|
|
- generated_from_trainer
|
|
base_model: Qwen/Qwen2.5-0.5B
|
|
datasets:
|
|
- Open-Orca/OpenOrca
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
---
|
|
|
|
# Model Card for Qwen2.5-0.5B-SFT-OpenOrca
|
|
|
|
This model is a fine-tuned version of [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) on a cleaned subset of the [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset. It was trained to follow instructions and reason across various tasks.
|
|
|
|
## Model Details
|
|
|
|
### Model Description
|
|
|
|
- **Model type:** Causal Language Model
|
|
- **Language(s) (NLP):** English
|
|
- **License:** Apache 2.0
|
|
- **Finetuned from model:** [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)
|
|
|
|
### Model Sources
|
|
|
|
- **Repository:** [Sepolian/qwen2.5-0.5b-sft-openorca](https://huggingface.co/Sepolian/qwen2.5-0.5b-sft-openorca)
|
|
- **Dataset:** [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca)
|
|
|
|
|
|
## Training Details
|
|
|
|
### Training Data
|
|
|
|
The model was trained on a **300k sample subset** of the OpenOrca dataset. We applied a rigorous **Data Cleaning Pipeline** to ensure high quality:
|
|
1. **Length & Refusal Filtering**: Removed samples with very short questions (<20 chars) or responses (<120 chars), and filtered out "I cannot answer..." refusals.
|
|
2. **Language Filtering**: Kept only English samples using a fast ASCII-ratio heuristic (>0.95) and `langdetect` fallback.
|
|
3. **Repetition Filtering**: Removed samples with excessive word repetition (consecutive words >5) or high n-gram frequency.
|
|
4. **MinHash Deduplication**: Applied 3-gram MinHash LSH (threshold 0.85) to remove near-duplicates.
|
|
5. **System Prompt Standardization**: Unified system prompts to a standard "You are a helpful assistant..." format.
|
|
|
|
### Training Procedure
|
|
|
|
#### Training Hyperparameters
|
|
|
|
- **Training regime:** bf16 mixed precision
|
|
- **Optimizer:** AdamW (`paged_adamw_32bit`)
|
|
- **Learning Rate:** 8e-6 with cosine schedule (warmup ratio 0.03)
|
|
- **Epochs:** 1
|
|
- **Packing:** Enabled (packed short sequences for efficiency)
|
|
|
|
#### Speeds, Sizes, Times
|
|
|
|
- **Hardware:** trained on NVIDIA L4 (24GB) via Google Colab
|
|
- **Training Time:** ~5 hours (approx)
|
|
|
|
## Evaluation
|
|
|
|
### Benchmarks
|
|
|
|
evaluated the model on standard benchmarks using `lm-evaluation-harness`.
|
|
|
|
|
|
|
|
|
|
## How to Get Started with the Model
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "Sepolian/qwen2.5-0.5b-sft-openorca"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Explain quantum computing in simple terms."},
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=False,
|
|
add_generation_prompt=True
|
|
)
|
|
|
|
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
|
|
|
generated_ids = model.generate(
|
|
model_inputs.input_ids,
|
|
max_new_tokens=512
|
|
)
|
|
|
|
generated_ids = [
|
|
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
|
|
]
|
|
|
|
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
|
print(response)
|
|
```
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@misc{openorca,
|
|
title = {OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces},
|
|
author = {Mukherjee, Subhabrata and Mitra, Arindam and Skjellum, Ganesh and Catalyurek, Umit and Tielelman, Thomas and Monroy-Hernandez, Andres},
|
|
year = {2023},
|
|
publisher = {HuggingFace}
|
|
}
|
|
```
|