---
library_name: transformers
tags:
  - qwen2.5
  - sft
  - openorca
  - generated_from_trainer
base_model: Qwen/Qwen2.5-0.5B
datasets:
  - Open-Orca/OpenOrca
language:
  - en
license: apache-2.0
---

# Model Card for Qwen2.5-0.5B-SFT-OpenOrca

This model is a fine-tuned version of [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) on a cleaned subset of the [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset. It was trained to follow instructions and reason across various tasks.

## Model Details

### Model Description

- **Model type:** Causal Language Model
- **Language(s) (NLP):** English
- **License:** Apache 2.0
- **Finetuned from model:** [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)

### Model Sources

- **Repository:** [Sepolian/qwen2.5-0.5b-sft-openorca](https://huggingface.co/Sepolian/qwen2.5-0.5b-sft-openorca)
- **Dataset:** [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca)


## Training Details

### Training Data

The model was trained on a **300k sample subset** of the OpenOrca dataset. We applied a rigorous **Data Cleaning Pipeline** to ensure high quality:
1.  **Length & Refusal Filtering**: Removed samples with very short questions (<20 chars) or responses (<120 chars), and filtered out "I cannot answer..." refusals.
2.  **Language Filtering**: Kept only English samples using a fast ASCII-ratio heuristic (>0.95) and `langdetect` fallback.
3.  **Repetition Filtering**: Removed samples with excessive word repetition (consecutive words >5) or high n-gram frequency.
4.  **MinHash Deduplication**: Applied 3-gram MinHash LSH (threshold 0.85) to remove near-duplicates.
5.  **System Prompt Standardization**: Unified system prompts to a standard "You are a helpful assistant..." format.

### Training Procedure

#### Training Hyperparameters

- **Training regime:** bf16 mixed precision
- **Optimizer:** AdamW (`paged_adamw_32bit`)
- **Learning Rate:** 8e-6 with cosine schedule (warmup ratio 0.03)
- **Epochs:** 1
- **Packing:** Enabled (packed short sequences for efficiency)

#### Speeds, Sizes, Times

- **Hardware:** trained on NVIDIA L4 (24GB) via Google Colab
- **Training Time:** ~5 hours (approx)

## Evaluation

### Benchmarks

evaluated the model on standard benchmarks using `lm-evaluation-harness`.


![full_eval](https://cdn-uploads.huggingface.co/production/uploads/67c90770fab614ccfdc7a6f5/wpDlF09G1rY75NtLdysyn.png)

## How to Get Started with the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Sepolian/qwen2.5-0.5b-sft-openorca"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)

generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```

## Citation

```bibtex
@misc{openorca,
  title = {OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces},
  author = {Mukherjee, Subhabrata and Mitra, Arindam and Skjellum, Ganesh and Catalyurek, Umit and Tielelman, Thomas and Monroy-Hernandez, Andres},
  year = {2023},
  publisher = {HuggingFace}
}
```