qwen2.5-coder-3b-distilled-from-14b-merged/README.md

language, license, base_model, tags, datasets, pipeline_tag

language	license	base_model	tags	datasets	pipeline_tag
en	apache-2.0	Qwen/Qwen2.5-Coder-3B-Instruct	qwen2.5 code knowledge-distillation gkd qlora python causal-lm	iamtarun/python_code_instructions_18k_alpaca	text-generation

Qwen2.5-Coder-3B — GKD Distilled from 14B

A merged (LoRA-free) version of Qwen2.5-Coder-3B-Instruct whose weights have been updated via Generalized Knowledge Distillation (GKD) from Qwen2.5-Coder-14B-Instruct as the teacher.

The LoRA adapter was trained with TRL's DistillationTrainer on a single NVIDIA A100 80 GB (Google Colab) for ~5 hours, then merged back into the base weights so the model loads exactly like the original 3B — no adapter plumbing required.

---

Evaluation — HumanEval pass@1

Greedy decoding, 164 Python programming tasks.

Model	Passed	Total	pass@1
Base Qwen2.5-Coder-3B-Instruct	133	164	81.1%
This model (distilled, 300 steps)	137	164	83.5%
Δ	+4	—	+2.44 pp

Task-level breakdown:

Outcome	Count
Both pass	126
Only distilled passes (gained)	11
Only base passes (lost)	7
Neither passes	20
Net change	+4

Gained tasks (11): HumanEval/26, /64, /75, /89, /93, /95, /110, /123, /124, /125, /135 — reasoning-heavy string manipulation, cipher encoding, and date parsing problems.

Lost tasks (7): HumanEval/10, /46, /99, /103, /141, /154, /159 — precise numeric/sequence edge cases (bankers rounding, fib4 base cases, cyclic rotation).

Note: HumanEval pass@1 with greedy decoding has ~±2pp noise. The +2.44pp delta is at the edge of statistical significance for a 300-step run; longer training is expected to widen the gap.

---

Training Details

Setup

Component	Detail
Teacher	Qwen/Qwen2.5-Coder-14B-Instruct — frozen, loaded in 4-bit NF4
Student	Qwen/Qwen2.5-Coder-3B-Instruct + LoRA (r=16, α=32)
Distillation	TRL DistillationTrainer, GKD with λ=0.25, β=0.5
Dataset	iamtarun/python_code_instructions_18k_alpaca — 17,681 train / 931 eval
Hardware	Google Colab A100 80 GB
Wall time	~5 hours

GKD Loss

L = λ · L_on_policy + (1−λ) · L_teacher_forced
    where L = β · KL(student ∥ teacher) + (1−β) · KL(teacher ∥ student)

• λ=0.25 — 25% on-policy (student-generated) sequences, 75% teacher-forced; keeps training stable at low step counts while reducing exposure bias.
• β=0.5 — symmetric Jensen–Shannon divergence between teacher and student logits.

Hyperparameters

Hyperparameter	Value
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q/k/v/o_proj, gate/up/down_proj
Optimizer	paged_adamw_8bit
Learning rate	1.5e-4
LR schedule	Cosine with 30-step warmup
Max steps	300
Per-device batch size	16
Gradient accumulation	2 (effective batch = 32)
Max sequence length	1024
Precision	bf16
Quantization (both models)	4-bit NF4, double quant, bf16 compute

---

Usage

This is a standard causal-LM checkpoint — load it exactly like the base model.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Harsha901/qwen2.5-coder-3b-distilled-from-14b-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Write a Python function that checks if a number is prime."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
    )

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Memory requirements

Precision	Approx. VRAM
bf16 (this checkpoint)	~6.5 GB
4-bit NF4 (bitsandbytes)	~2.5 GB

The model runs comfortably on a 24 GB consumer GPU (RTX 3090/4090) in full bf16 precision.

---

Limitations and Future Work

• Only 300 steps trained — the model has seen ~9,600 examples (~0.54 epochs of the training set). Longer training (1,000–2,000 steps or 3 full epochs) is expected to improve further.
• Regressions on arithmetic edge cases — the 7 lost tasks suggest slight distribution shift away from precise numeric corner cases; increasing LoRA rank or adding targeted examples may help.
• Training dataset is Python-only — generalisation to other languages is untested.
• Distillation dataset differs from eval benchmark — HumanEval is held-out; the training set is python_code_instructions_18k_alpaca, which covers general Python instruction-following rather than competitive algorithmic problems.

Planned next steps:

1. Train for 1,000+ steps and track HumanEval every 50 steps.
2. Tune λ toward 0.5–0.75 for more on-policy exposure.
3. Benchmark inference latency and VRAM vs. the 14B teacher to quantify serving cost reduction.

---

Related Resources

• LoRA adapter (pre-merge): Harsha901/qwen2.5-coder-3b-distilled-from-14b
• Base student model: Qwen/Qwen2.5-Coder-3B-Instruct
• Teacher model: Qwen/Qwen2.5-Coder-14B-Instruct
• Training dataset: iamtarun/python_code_instructions_18k_alpaca
• TRL DistillationTrainer docs: huggingface.co/docs/trl