初始化项目，由ModelHub XC社区提供模型
Model: Henry665/Walkie-Code-0.5B Source: Original Platform
This commit is contained in:
ModelHub XC
153
README.md
Normal file
153
README.md
Normal file
@@ -0,0 +1,153 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- code
|
||||
tags:
|
||||
- code
|
||||
- python
|
||||
- causal-lm
|
||||
- walkie
|
||||
- dapo
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Walkie-Code-0.5B
|
||||
|
||||
**Walkie-Code-0.5B** is a ~501M parameter decoder-only language model focused on **Python code generation**. It is the flagship model from the [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through) project — an educational and engineering effort to walk through modern LLM design from modular components to a full training pipeline.
|
||||
|
||||
This checkpoint is the **DAPO RL best** model after: **17B-token pretraining → KodCode SFT → DAPO reinforcement learning**.
|
||||
|
||||
> **Note on config format:** weights are exported in a Qwen3-compatible layout (`model_type: qwen3`) because Walkie shares the same structural primitives (RMSNorm, SwiGLU, GQA, QK-Norm). This is an export convention for Transformers/vLLM compatibility — the model is **Walkie-Code-0.5B**, not an official Qwen release.
|
||||
|
||||
## Model Summary
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Parameters | ~501M |
|
||||
| Architecture | 24-layer decoder, hidden 1280, GQA 4:1 (20Q / 5KV) |
|
||||
| Context length | 4096 tokens |
|
||||
| Vocab size | 65536 (BPE) |
|
||||
| FFN | SwiGLU, d_ffn=3456 |
|
||||
| Position encoding | RoPE (θ=5×10⁵) |
|
||||
| Training stages | Pretrain → SFT → DAPO RL |
|
||||
| Best RL method | DAPO (Direct Advantage Policy Optimization) |
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
Evaluated on HumanEval, HumanEval+, MBPP, MBPP+ with **n=8**, temperature=0.2, top_p=0.95, sandbox execution.
|
||||
|
||||
### Macro average (this checkpoint)
|
||||
|
||||
| Metric | Score |
|
||||
|--------|------:|
|
||||
| pass@1 | **37.6%** |
|
||||
| pass@4 | **43.6%** |
|
||||
| pass@8 | **46.6%** |
|
||||
|
||||
### Per-dataset pass@1
|
||||
|
||||
| Dataset | pass@1 |
|
||||
|---------|-------:|
|
||||
| HumanEval+ | 34.1% |
|
||||
| HumanEval | 36.1% |
|
||||
| MBPP | 42.1% |
|
||||
| MBPP+ | 38.3% |
|
||||
|
||||
### Comparison vs Qwen2.5-0.5B-Instruct (pass@1 macro)
|
||||
|
||||
| Model | Macro pass@1 |
|
||||
|-------|-------------:|
|
||||
| Qwen2.5-0.5B-Instruct | 36.7% |
|
||||
| **Walkie-Code-0.5B (this)** | **38.4%** (+1.7 pp) |
|
||||
|
||||
### Full training pipeline (macro pass@1)
|
||||
|
||||
| Stage | pass@1 | pass@8 |
|
||||
|-------|-------:|-------:|
|
||||
| SFT | 33.7% | 41.4% |
|
||||
| DAPO (this checkpoint) | 37.6% | 46.6% |
|
||||
|
||||
## Training Details
|
||||
|
||||
**Pretraining (~17B tokens, code-heavy ~70%)**
|
||||
- The Stack v2 Python, StarCoder Python Edu, FineWeb Edu, FineMath, OPC Annealing
|
||||
- Two-stage WSD schedule (main 89% + anneal 11%)
|
||||
- Muon + AdamW mixed optimizer, FlashAttention-2
|
||||
|
||||
**SFT**
|
||||
- KodCode-V1-SFT-R1 (~246k samples), DeepSeek-R1 generated solutions
|
||||
- Instruct/complete Python function generation, ~2.6 epochs
|
||||
|
||||
**RL (DAPO)**
|
||||
- KodCode-V1-RL (~12k filtered samples)
|
||||
- Online rollout with code sandbox rewards (pass/fail)
|
||||
- Dynamic group filtering + clipped policy gradient (ε_l=0.2, ε_h=0.28)
|
||||
|
||||
## Usage
|
||||
|
||||
### Transformers
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_id = "Henry665/Walkie-Code-0.5B"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
prompt = "user: Write a Python function to check if a number is prime.\nassistant:"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2, top_p=0.95)
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
### vLLM
|
||||
|
||||
```bash
|
||||
vllm serve Henry665/Walkie-Code-0.5B --dtype auto --max-model-len 4096
|
||||
```
|
||||
|
||||
### Prompt format
|
||||
|
||||
Training used a simple dialog template:
|
||||
|
||||
```
|
||||
user: <instruction>
|
||||
assistant: <python code>
|
||||
```
|
||||
|
||||
For code benchmarks, plain `user:` / `assistant:` text prompts are recommended.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Specialized for **Python code generation**; general chat / multilingual ability is limited.
|
||||
- Small scale (0.5B); not competitive with much larger models on broad reasoning.
|
||||
- Exported as Qwen3-compatible config for tooling — verify behavior matches Walkie training setup.
|
||||
- Benchmark scores depend on prompt template, sandbox, and sampling settings.
|
||||
|
||||
## Citation & Links
|
||||
|
||||
- Project: [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through)
|
||||
- Architecture: RMSNorm, RoPE, GQA, SwiGLU, QK-Norm, Muon optimizer
|
||||
- RL method: DAPO (dynamic filtering + clipped surrogate)
|
||||
|
||||
```bibtex
|
||||
@misc{walkie-code-0.5b,
|
||||
title={Walkie-Code-0.5B: A Modular 0.5B Python Code LLM},
|
||||
author={LLM Walk-Through Team},
|
||||
year={2026},
|
||||
howpublished={\url{https://huggingface.co/Henry665/Walkie-Code-0.5B}}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
Reference in New Issue
Block a user