Walkie-Code-0.5B/README.md

---
license: apache-2.0
language:
- en
- code
tags:
- code
- python
- causal-lm
- walkie
- dapo
library_name: transformers
pipeline_tag: text-generation
---

# Walkie-Code-0.5B

**Walkie-Code-0.5B** is a ~501M parameter decoder-only language model focused on **Python code generation**. It is the flagship model from the [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through) project — an educational and engineering effort to walk through modern LLM design from modular components to a full training pipeline.

This checkpoint is the **DAPO RL best** model after: **17B-token pretraining → KodCode SFT → DAPO reinforcement learning**.

> **Note on config format:** weights are exported in a Qwen3-compatible layout (`model_type: qwen3`) because Walkie shares the same structural primitives (RMSNorm, SwiGLU, GQA, QK-Norm). This is an export convention for Transformers/vLLM compatibility — the model is **Walkie-Code-0.5B**, not an official Qwen release.

## Model Summary

| Item | Value |
|------|-------|
| Parameters | ~501M |
| Architecture | 24-layer decoder, hidden 1280, GQA 4:1 (20Q / 5KV) |
| Context length | 4096 tokens |
| Vocab size | 65536 (BPE) |
| FFN | SwiGLU, d_ffn=3456 |
| Position encoding | RoPE (θ=5×10⁵) |
| Training stages | Pretrain → SFT → DAPO RL |
| Best RL method | DAPO (Direct Advantage Policy Optimization) |

## Benchmark Results

Evaluated on HumanEval, HumanEval+, MBPP, MBPP+ with **n=8**, temperature=0.2, top_p=0.95, sandbox execution.

### Macro average (this checkpoint)

| Metric | Score |
|--------|------:|
| pass@1 | **37.6%** |
| pass@4 | **43.6%** |
| pass@8 | **46.6%** |

### Per-dataset pass@1

| Dataset | pass@1 |
|---------|-------:|
| HumanEval+ | 34.1% |
| HumanEval | 36.1% |
| MBPP | 42.1% |
| MBPP+ | 38.3% |

### Comparison vs Qwen2.5-0.5B-Instruct (pass@1 macro)

| Model | Macro pass@1 |
|-------|-------------:|
| Qwen2.5-0.5B-Instruct | 36.7% |
| **Walkie-Code-0.5B (this)** | **38.4%** (+1.7 pp) |

### Full training pipeline (macro pass@1)

| Stage | pass@1 | pass@8 |
|-------|-------:|-------:|
| SFT | 33.7% | 41.4% |
| DAPO (this checkpoint) | 37.6% | 46.6% |

## Training Details

**Pretraining (~17B tokens, code-heavy ~70%)**
- The Stack v2 Python, StarCoder Python Edu, FineWeb Edu, FineMath, OPC Annealing
- Two-stage WSD schedule (main 89% + anneal 11%)
- Muon + AdamW mixed optimizer, FlashAttention-2

**SFT**
- KodCode-V1-SFT-R1 (~246k samples), DeepSeek-R1 generated solutions
- Instruct/complete Python function generation, ~2.6 epochs

**RL (DAPO)**
- KodCode-V1-RL (~12k filtered samples)
- Online rollout with code sandbox rewards (pass/fail)
- Dynamic group filtering + clipped policy gradient (ε_l=0.2, ε_h=0.28)

## Usage

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Henry665/Walkie-Code-0.5B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "user: Write a Python function to check if a number is prime.\nassistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### vLLM

```bash
vllm serve Henry665/Walkie-Code-0.5B --dtype auto --max-model-len 4096
```

### Prompt format

Training used a simple dialog template:

```
user: <instruction>
assistant: <python code>
```

For code benchmarks, plain `user:` / `assistant:` text prompts are recommended.

## Limitations

- Specialized for **Python code generation**; general chat / multilingual ability is limited.
- Small scale (0.5B); not competitive with much larger models on broad reasoning.
- Exported as Qwen3-compatible config for tooling — verify behavior matches Walkie training setup.
- Benchmark scores depend on prompt template, sandbox, and sampling settings.

## Citation & Links

- Project: [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through)
- Architecture: RMSNorm, RoPE, GQA, SwiGLU, QK-Norm, Muon optimizer
- RL method: DAPO (dynamic filtering + clipped surrogate)

```bibtex
@misc{walkie-code-0.5b,
  title={Walkie-Code-0.5B: A Modular 0.5B Python Code LLM},
  author={LLM Walk-Through Team},
  year={2026},
  howpublished={\url{https://huggingface.co/Henry665/Walkie-Code-0.5B}}
}
```

## License

Apache 2.0