初始化项目，由ModelHub XC社区提供模型

Model: Henry665/Walkie-Code-0.5B
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,153 @@
+---
+license: apache-2.0
+language:
+- en
+- code
+tags:
+- code
+- python
+- causal-lm
+- walkie
+- dapo
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# Walkie-Code-0.5B
+
+**Walkie-Code-0.5B** is a ~501M parameter decoder-only language model focused on **Python code generation**. It is the flagship model from the [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through) project — an educational and engineering effort to walk through modern LLM design from modular components to a full training pipeline.
+
+This checkpoint is the **DAPO RL best** model after: **17B-token pretraining → KodCode SFT → DAPO reinforcement learning**.
+
+> **Note on config format:** weights are exported in a Qwen3-compatible layout (`model_type: qwen3`) because Walkie shares the same structural primitives (RMSNorm, SwiGLU, GQA, QK-Norm). This is an export convention for Transformers/vLLM compatibility — the model is **Walkie-Code-0.5B**, not an official Qwen release.
+
+## Model Summary
+
+| Item | Value |
+|------|-------|
+| Parameters | ~501M |
+| Architecture | 24-layer decoder, hidden 1280, GQA 4:1 (20Q / 5KV) |
+| Context length | 4096 tokens |
+| Vocab size | 65536 (BPE) |
+| FFN | SwiGLU, d_ffn=3456 |
+| Position encoding | RoPE (θ=5×10⁵) |
+| Training stages | Pretrain → SFT → DAPO RL |
+| Best RL method | DAPO (Direct Advantage Policy Optimization) |
+
+## Benchmark Results
+
+Evaluated on HumanEval, HumanEval+, MBPP, MBPP+ with **n=8**, temperature=0.2, top_p=0.95, sandbox execution.
+
+### Macro average (this checkpoint)
+
+| Metric | Score |
+|--------|------:|
+| pass@1 | **37.6%** |
+| pass@4 | **43.6%** |
+| pass@8 | **46.6%** |
+
+### Per-dataset pass@1
+
+| Dataset | pass@1 |
+|---------|-------:|
+| HumanEval+ | 34.1% |
+| HumanEval | 36.1% |
+| MBPP | 42.1% |
+| MBPP+ | 38.3% |
+
+### Comparison vs Qwen2.5-0.5B-Instruct (pass@1 macro)
+
+| Model | Macro pass@1 |
+|-------|-------------:|
+| Qwen2.5-0.5B-Instruct | 36.7% |
+| **Walkie-Code-0.5B (this)** | **38.4%** (+1.7 pp) |
+
+### Full training pipeline (macro pass@1)
+
+| Stage | pass@1 | pass@8 |
+|-------|-------:|-------:|
+| SFT | 33.7% | 41.4% |
+| DAPO (this checkpoint) | 37.6% | 46.6% |
+
+## Training Details
+
+**Pretraining (~17B tokens, code-heavy ~70%)**
+- The Stack v2 Python, StarCoder Python Edu, FineWeb Edu, FineMath, OPC Annealing
+- Two-stage WSD schedule (main 89% + anneal 11%)
+- Muon + AdamW mixed optimizer, FlashAttention-2
+
+**SFT**
+- KodCode-V1-SFT-R1 (~246k samples), DeepSeek-R1 generated solutions
+- Instruct/complete Python function generation, ~2.6 epochs
+
+**RL (DAPO)**
+- KodCode-V1-RL (~12k filtered samples)
+- Online rollout with code sandbox rewards (pass/fail)
+- Dynamic group filtering + clipped policy gradient (ε_l=0.2, ε_h=0.28)
+
+## Usage
+
+### Transformers
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "Henry665/Walkie-Code-0.5B"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+prompt = "user: Write a Python function to check if a number is prime.\nassistant:"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2, top_p=0.95)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+### vLLM
+
+```bash
+vllm serve Henry665/Walkie-Code-0.5B --dtype auto --max-model-len 4096
+```
+
+### Prompt format
+
+Training used a simple dialog template:
+
+```
+user: <instruction>
+assistant: <python code>
+```
+
+For code benchmarks, plain `user:` / `assistant:` text prompts are recommended.
+
+## Limitations
+
+- Specialized for **Python code generation**; general chat / multilingual ability is limited.
+- Small scale (0.5B); not competitive with much larger models on broad reasoning.
+- Exported as Qwen3-compatible config for tooling — verify behavior matches Walkie training setup.
+- Benchmark scores depend on prompt template, sandbox, and sampling settings.
+
+## Citation & Links
+
+- Project: [LLM Walk-Through](https://github.com/HenryNotTheKing/LLM-Walk-Through)
+- Architecture: RMSNorm, RoPE, GQA, SwiGLU, QK-Norm, Muon optimizer
+- RL method: DAPO (dynamic filtering + clipped surrogate)
+
+```bibtex
+@misc{walkie-code-0.5b,
+  title={Walkie-Code-0.5B: A Modular 0.5B Python Code LLM},
+  author={LLM Walk-Through Team},
+  year={2026},
+  howpublished={\url{https://huggingface.co/Henry665/Walkie-Code-0.5B}}
+}
+```
+
+## License
+
+Apache 2.0

config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "model_type": "qwen3",
+  "vocab_size": 65536,
+  "hidden_size": 1280,
+  "intermediate_size": 3456,
+  "num_hidden_layers": 24,
+  "num_attention_heads": 20,
+  "num_key_value_heads": 5,
+  "head_dim": 64,
+  "max_position_embeddings": 4096,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 500000.0,
+  "attention_bias": false,
+  "tie_word_embeddings": true,
+  "hidden_act": "silu",
+  "torch_dtype": "bfloat16",
+  "bos_token_id": null,
+  "eos_token_id": 0,
+  "pad_token_id": 1,
+  "use_cache": true
+}

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "bos_token_id": null,
+  "eos_token_id": 0,
+  "pad_token_id": 1
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92c0fe7bb0fe990e8a0c39eb8e5adc23627797146de04054fa1a8c61987c04ee
+size 2338617896