qwen2.5-coder-3b-distilled-from-14b-merged/README.md

---
language:
- en
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-3B-Instruct
tags:
- qwen2.5
- code
- knowledge-distillation
- gkd
- qlora
- python
- causal-lm
datasets:
- iamtarun/python_code_instructions_18k_alpaca
pipeline_tag: text-generation
---

# Qwen2.5-Coder-3B — GKD Distilled from 14B

A **merged** (LoRA-free) version of `Qwen2.5-Coder-3B-Instruct` whose weights have been updated via **Generalized Knowledge Distillation (GKD)** from `Qwen2.5-Coder-14B-Instruct` as the teacher.

The LoRA adapter was trained with TRL's `DistillationTrainer` on a single NVIDIA A100 80 GB (Google Colab) for ~5 hours, then merged back into the base weights so the model loads exactly like the original 3B — no adapter plumbing required.

---

## Evaluation — HumanEval pass@1

Greedy decoding, 164 Python programming tasks.

| Model | Passed | Total | pass@1 |
|---|---:|---:|---:|
| Base `Qwen2.5-Coder-3B-Instruct` | 133 | 164 | 81.1% |
| **This model (distilled, 300 steps)** | **137** | **164** | **83.5%** |
| Δ | +4 | — | **+2.44 pp** |

**Task-level breakdown:**

| Outcome | Count |
|---|---:|
| Both pass | 126 |
| Only distilled passes (gained) | 11 |
| Only base passes (lost) | 7 |
| Neither passes | 20 |
| **Net change** | **+4** |

**Gained tasks (11):** HumanEval/26, /64, /75, /89, /93, /95, /110, /123, /124, /125, /135 — reasoning-heavy string manipulation, cipher encoding, and date parsing problems.

**Lost tasks (7):** HumanEval/10, /46, /99, /103, /141, /154, /159 — precise numeric/sequence edge cases (bankers rounding, fib4 base cases, cyclic rotation).

> Note: HumanEval pass@1 with greedy decoding has ~±2pp noise. The +2.44pp delta is at the edge of statistical significance for a 300-step run; longer training is expected to widen the gap.

---

## Training Details

### Setup

| Component | Detail |
|---|---|
| **Teacher** | `Qwen/Qwen2.5-Coder-14B-Instruct` — frozen, loaded in 4-bit NF4 |
| **Student** | `Qwen/Qwen2.5-Coder-3B-Instruct` + LoRA (r=16, α=32) |
| **Distillation** | TRL `DistillationTrainer`, GKD with λ=0.25, β=0.5 |
| **Dataset** | `iamtarun/python_code_instructions_18k_alpaca` — 17,681 train / 931 eval |
| **Hardware** | Google Colab A100 80 GB |
| **Wall time** | ~5 hours |

### GKD Loss

```
L = λ · L_on_policy + (1−λ) · L_teacher_forced
    where L = β · KL(student ∥ teacher) + (1−β) · KL(teacher ∥ student)
```

- **λ=0.25** — 25% on-policy (student-generated) sequences, 75% teacher-forced; keeps training stable at low step counts while reducing exposure bias.
- **β=0.5** — symmetric Jensen–Shannon divergence between teacher and student logits.

### Hyperparameters

| Hyperparameter | Value |
|---|---|
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Target modules | q/k/v/o_proj, gate/up/down_proj |
| Optimizer | paged_adamw_8bit |
| Learning rate | 1.5e-4 |
| LR schedule | Cosine with 30-step warmup |
| Max steps | 300 |
| Per-device batch size | 16 |
| Gradient accumulation | 2 (effective batch = 32) |
| Max sequence length | 1024 |
| Precision | bf16 |
| Quantization (both models) | 4-bit NF4, double quant, bf16 compute |

---

## Usage

This is a standard causal-LM checkpoint — load it exactly like the base model.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Harsha901/qwen2.5-coder-3b-distilled-from-14b-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Write a Python function that checks if a number is prime."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
    )

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### Memory requirements

| Precision | Approx. VRAM |
|---|---|
| bf16 (this checkpoint) | ~6.5 GB |
| 4-bit NF4 (bitsandbytes) | ~2.5 GB |

The model runs comfortably on a 24 GB consumer GPU (RTX 3090/4090) in full bf16 precision.

---

## Limitations and Future Work

- **Only 300 steps trained** — the model has seen ~9,600 examples (~0.54 epochs of the training set). Longer training (1,000–2,000 steps or 3 full epochs) is expected to improve further.
- **Regressions on arithmetic edge cases** — the 7 lost tasks suggest slight distribution shift away from precise numeric corner cases; increasing LoRA rank or adding targeted examples may help.
- **Training dataset is Python-only** — generalisation to other languages is untested.
- **Distillation dataset differs from eval benchmark** — HumanEval is held-out; the training set is `python_code_instructions_18k_alpaca`, which covers general Python instruction-following rather than competitive algorithmic problems.

Planned next steps:
1. Train for 1,000+ steps and track HumanEval every 50 steps.
2. Tune λ toward 0.5–0.75 for more on-policy exposure.
3. Benchmark inference latency and VRAM vs. the 14B teacher to quantify serving cost reduction.

---

## Related Resources

- **LoRA adapter (pre-merge):** [Harsha901/qwen2.5-coder-3b-distilled-from-14b](https://huggingface.co/Harsha901/qwen2.5-coder-3b-distilled-from-14b)
- **Base student model:** [Qwen/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct)
- **Teacher model:** [Qwen/Qwen2.5-Coder-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct)
- **Training dataset:** [iamtarun/python_code_instructions_18k_alpaca](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca)
- **TRL DistillationTrainer docs:** [huggingface.co/docs/trl](https://huggingface.co/docs/trl)