154 lines
4.4 KiB
Markdown
154 lines
4.4 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
- code
|
||
tags:
|
||
- code
|
||
- python
|
||
- causal-lm
|
||
- walkie
|
||
- dapo
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Walkie-Code-0.5B
|
||
|
||
**Walkie-Code-0.5B** is a ~501M parameter decoder-only language model focused on **Python code generation**. It is the flagship model from the [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through) project — an educational and engineering effort to walk through modern LLM design from modular components to a full training pipeline.
|
||
|
||
This checkpoint is the **DAPO RL best** model after: **17B-token pretraining → KodCode SFT → DAPO reinforcement learning**.
|
||
|
||
> **Note on config format:** weights are exported in a Qwen3-compatible layout (`model_type: qwen3`) because Walkie shares the same structural primitives (RMSNorm, SwiGLU, GQA, QK-Norm). This is an export convention for Transformers/vLLM compatibility — the model is **Walkie-Code-0.5B**, not an official Qwen release.
|
||
|
||
## Model Summary
|
||
|
||
| Item | Value |
|
||
|------|-------|
|
||
| Parameters | ~501M |
|
||
| Architecture | 24-layer decoder, hidden 1280, GQA 4:1 (20Q / 5KV) |
|
||
| Context length | 4096 tokens |
|
||
| Vocab size | 65536 (BPE) |
|
||
| FFN | SwiGLU, d_ffn=3456 |
|
||
| Position encoding | RoPE (θ=5×10⁵) |
|
||
| Training stages | Pretrain → SFT → DAPO RL |
|
||
| Best RL method | DAPO (Direct Advantage Policy Optimization) |
|
||
|
||
## Benchmark Results
|
||
|
||
Evaluated on HumanEval, HumanEval+, MBPP, MBPP+ with **n=8**, temperature=0.2, top_p=0.95, sandbox execution.
|
||
|
||
### Macro average (this checkpoint)
|
||
|
||
| Metric | Score |
|
||
|--------|------:|
|
||
| pass@1 | **37.6%** |
|
||
| pass@4 | **43.6%** |
|
||
| pass@8 | **46.6%** |
|
||
|
||
### Per-dataset pass@1
|
||
|
||
| Dataset | pass@1 |
|
||
|---------|-------:|
|
||
| HumanEval+ | 34.1% |
|
||
| HumanEval | 36.1% |
|
||
| MBPP | 42.1% |
|
||
| MBPP+ | 38.3% |
|
||
|
||
### Comparison vs Qwen2.5-0.5B-Instruct (pass@1 macro)
|
||
|
||
| Model | Macro pass@1 |
|
||
|-------|-------------:|
|
||
| Qwen2.5-0.5B-Instruct | 36.7% |
|
||
| **Walkie-Code-0.5B (this)** | **38.4%** (+1.7 pp) |
|
||
|
||
### Full training pipeline (macro pass@1)
|
||
|
||
| Stage | pass@1 | pass@8 |
|
||
|-------|-------:|-------:|
|
||
| SFT | 33.7% | 41.4% |
|
||
| DAPO (this checkpoint) | 37.6% | 46.6% |
|
||
|
||
## Training Details
|
||
|
||
**Pretraining (~17B tokens, code-heavy ~70%)**
|
||
- The Stack v2 Python, StarCoder Python Edu, FineWeb Edu, FineMath, OPC Annealing
|
||
- Two-stage WSD schedule (main 89% + anneal 11%)
|
||
- Muon + AdamW mixed optimizer, FlashAttention-2
|
||
|
||
**SFT**
|
||
- KodCode-V1-SFT-R1 (~246k samples), DeepSeek-R1 generated solutions
|
||
- Instruct/complete Python function generation, ~2.6 epochs
|
||
|
||
**RL (DAPO)**
|
||
- KodCode-V1-RL (~12k filtered samples)
|
||
- Online rollout with code sandbox rewards (pass/fail)
|
||
- Dynamic group filtering + clipped policy gradient (ε_l=0.2, ε_h=0.28)
|
||
|
||
## Usage
|
||
|
||
### Transformers
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_id = "Henry665/Walkie-Code-0.5B"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
trust_remote_code=True,
|
||
)
|
||
|
||
prompt = "user: Write a Python function to check if a number is prime.\nassistant:"
|
||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2, top_p=0.95)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
### vLLM
|
||
|
||
```bash
|
||
vllm serve Henry665/Walkie-Code-0.5B --dtype auto --max-model-len 4096
|
||
```
|
||
|
||
### Prompt format
|
||
|
||
Training used a simple dialog template:
|
||
|
||
```
|
||
user: <instruction>
|
||
assistant: <python code>
|
||
```
|
||
|
||
For code benchmarks, plain `user:` / `assistant:` text prompts are recommended.
|
||
|
||
## Limitations
|
||
|
||
- Specialized for **Python code generation**; general chat / multilingual ability is limited.
|
||
- Small scale (0.5B); not competitive with much larger models on broad reasoning.
|
||
- Exported as Qwen3-compatible config for tooling — verify behavior matches Walkie training setup.
|
||
- Benchmark scores depend on prompt template, sandbox, and sampling settings.
|
||
|
||
## Citation & Links
|
||
|
||
- Project: [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through)
|
||
- Architecture: RMSNorm, RoPE, GQA, SwiGLU, QK-Norm, Muon optimizer
|
||
- RL method: DAPO (dynamic filtering + clipped surrogate)
|
||
|
||
```bibtex
|
||
@misc{walkie-code-0.5b,
|
||
title={Walkie-Code-0.5B: A Modular 0.5B Python Code LLM},
|
||
author={LLM Walk-Through Team},
|
||
year={2026},
|
||
howpublished={\url{https://huggingface.co/Henry665/Walkie-Code-0.5B}}
|
||
}
|
||
```
|
||
|
||
## License
|
||
|
||
Apache 2.0
|