---
license: apache-2.0
datasets:
- Jackrong/DeepSeek-V4-Distill-8000x
language:
- en
base_model:
- Qwen/Qwen2-0.5B-Instruct
pipeline_tag: text-generation
tags:
- slm
- trl
- text-generation-inference
- reasoning
- thinking
- chat
---

# Dragon-1.5-0.5B (qwen2-0.5b-reasoning v3.1.1)

![Dragon Logo](./dragon_logo_a.png)
                                                                                                                                
Dragon is a lightweight general reasoning model built upon the base [Qwen2-0.5B-instruct model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct). It offers accurate and quick text generation on a variety of topics(including code related problems).
It's small size (0.5B parameters) allows it to run comfortably on most laptop/commercial grade GPUs.
This model also offers Q/A and subject matter expert capabilities on general and code related subjects.

The Dragon-1.5 is the next generation for the [Dragon-1/1.5 series](https://huggingface.co/collections/DireDreadlord/dragon-1-15) which incorporates high-end reasoning capabilities into the standard Qwen2 architecture.

The 0.5B variant has been SFT trained on general/code reasoning traces found [here](https://huggingface.co/datasets/Jackrong/DeepSeek-V4-Distill-8000x) with further RL training carried out via. a GRPO algorithm. This endows the model with enhanced reasoning capabilities which allows it to serve higher quality and hallucination-free generations.

---


**Estimated parameters:** ~0.5B

**Architecture:** Qwen2

**Intended use:** Advanced reasoning, instruction following along with enhanced code snippet and long form code generation 

---


## Training data
**Phase-1**
- Source: deepseek-v4-distill-8000x dataset (https://huggingface.co/datasets/Jackrong/DeepSeek-V4-Distill-8000x)
- Rows: ~7,716 rows templated with a custom .jinja chat format
- Training: trained for 4,000 steps on an A10 (24GB VRAM)

**Phase-2**
- Source: deepseek-v4-reasoning-code-2500 dataset (https://huggingface.co/datasets/Banaxi-Tech/Deepseek-V4-Reasoning-Code-2500)
- Rows: ~7,716 rows templated with a custom .jinja chat format
- Training: trained via. GRPO for 350 steps on an A10 (24GB VRAM)


## Usage

Install requirements:

```bash
pip install -r requirements.txt
pip install transformers datasets accelerate safetensors
```


## Usage (Hugging Face Hub)
You can load it directly from HuggingFace:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer


device = "cuda" if torch.cuda.is_available() else "cpu"

model_id = "DireDreadlord/Dragon-1.5-0.5B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype="auto"
)
model.to(device)
tokenizer = AutoTokenizer.from_pretrained(model_id)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)

prompt = "Solve the leetcode problem 1: two sum using the hash map technique"

input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    return_tensors="pt",
    tokenize=True,
)["input_ids"].to(device)

output = model.generate(
    input_ids,
    do_sample=True,
    temperature=0.4,
    top_k=50,
    repetition_penalty=1.05,
    max_new_tokens=2048,
    streamer=streamer,
)
```
**For optimal long-form generation(with reasoning), set `max_new_tokens=2048`**

## Limitations
- Model for experimental use only; users should employ it as such under license.