277 lines
9.1 KiB
Markdown
277 lines
9.1 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
tags:
|
||
- llama
|
||
- causal-lm
|
||
- text-generation
|
||
- instruction-tuned
|
||
- nano
|
||
- v5.1
|
||
- pytorch
|
||
- sequence-packing
|
||
pipeline_tag: text-generation
|
||
datasets:
|
||
- Open-Orca/OpenOrca
|
||
- meta-math/MetaMathQA
|
||
- Roman1111111/claude-opus-4.6-10000x
|
||
- WizardLM/WizardLM_evol_instruct_V2_196k
|
||
- WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K
|
||
- microsoft/orca-math-word-problems-200k
|
||
- lighteval/MATH-Hard
|
||
- HuggingFaceH4/MATH-500
|
||
- garage-bAInd/Open-Platypus
|
||
- teknium/OpenHermes-2.5
|
||
- ise-uiuc/Magicoder-OSS-Instruct-75K
|
||
- m-a-p/CodeFeedback-Filtered-Instruction
|
||
- iamtarun/python_code_instructions_18k_alpaca
|
||
- nvidia/OpenCodeInstruct
|
||
- b-mc2/sql-create-context
|
||
- HuggingFaceH4/ultrachat_200k
|
||
- databricks/databricks-dolly-15k
|
||
- Amod/mental_health_counseling_conversations
|
||
- mlabonne/guanaco-llama2-1k
|
||
- ray0rf1re/FineWeb-Nano
|
||
- ray0rf1re/hyper-pip
|
||
- flytech/python-codes-25k
|
||
- ByteDance-Seed/Code-Contests-Plus
|
||
- open-thoughts/OpenThoughts-TB-dev
|
||
- Nix-ai/cat-math-v1
|
||
- Nix-ai/Cat-v2.8XXXL-plus
|
||
- ray0rf1re/claude1255x9
|
||
- ray0rf1re/archlinux-v1
|
||
- HuggingFaceFW/fineweb-edu
|
||
- ajibawa-2023/Code-74k-ShareGPT
|
||
- codeparrot/apps
|
||
- allenai/moral_stories
|
||
- mrm8488/shell-commands-prompts
|
||
model-index:
|
||
- name: Nano-Nano v5.1
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
metrics:
|
||
- name: Training Loss
|
||
type: loss
|
||
value: 2.0444
|
||
- name: Overall Eval Score
|
||
type: accuracy
|
||
value: 0.1667
|
||
- name: Knowledge
|
||
type: accuracy
|
||
value: 0
|
||
- name: Reasoning
|
||
type: accuracy
|
||
value: 0
|
||
- name: Hallucination Resistance
|
||
type: accuracy
|
||
value: 0
|
||
- name: Instruction Following
|
||
type: accuracy
|
||
value: 0.5
|
||
- name: Coherence
|
||
type: accuracy
|
||
value: 0.3333
|
||
---
|
||
|
||
<div align="center">
|
||
|
||
# 🧠 Nano-Nano v5.1
|
||
|
||
### ~1218.3 M · Qwen3 · 300M · GQA + QK-Norm · Sequence-Packed · 26 Datasets
|
||
|
||
[](https://opensource.org/licenses/Apache-2.0)
|
||
[](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
|
||
[](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
|
||
[](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
|
||
[](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
|
||
|
||
**Fully redesigned** successor to Nano-nano v4.5.
|
||
~298M Qwen3 parameters trained with **sequence packing** on a quality-tiered 34-dataset mix.
|
||
Features loss-boost system: auto-extends training if loss > 4.95 (up to 3×75 steps).
|
||
Goal: loss < 2.5 through compute efficiency, not raw scale.
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
## 📋 Summary
|
||
|
||
| | |
|
||
|---|---|
|
||
| Architecture | LLaMA decoder-only |
|
||
| Parameters | ~1218.3 M |
|
||
| Context | 2 048 tokens |
|
||
| Vocabulary | 50,304 tokens |
|
||
| Training loss | `2.0444` |
|
||
| Eval score | `16.7%` |
|
||
| Tokens trained | 0.01 B (sequence-packed) |
|
||
| Hardware | GTX 1080 8 GB (Pascal) |
|
||
|
||
---
|
||
|
||
## 🏗️ Architecture (v4 → v4.5 → v5.1)
|
||
|
||
| Hyperparameter | v4 | v4.5 | **v5.1** |
|
||
|---|---|---|---|
|
||
| Parameters | ~236 M | ~256 M | **~1218.3 M** (~1.218 B) |
|
||
| `hidden_size` | 896 | 896 | **1 024** |
|
||
| `intermediate_size` | 2 688 | 2 912 | **2 730** (8/3×hidden) |
|
||
| `num_hidden_layers` | 14 | 15 | **16** |
|
||
| `num_attention_heads` | 14 | 14 | **16** |
|
||
| `num_key_value_heads` | 14 | 14 | **16** |
|
||
| `head_dim` | 64 | 64 | 64 |
|
||
| `vocab_size` | 50 264 | 50 264 | 50,304 |
|
||
| `max_position_embeddings` | 1 024 | 2 048 | 2 048 |
|
||
| `rms_norm_eps` | 1e-6 | 1e-6 | **1e-5** |
|
||
| `rope_theta` | 10 000 | 10 000 | 10 000 |
|
||
| `rope_scaling` | — | linear 2× | linear 2× |
|
||
| `tie_word_embeddings` | False | False | False |
|
||
| Sequence packing | ❌ | ❌ | **✅ 1× packed** |
|
||
| Architecture | LLaMA | LLaMA | **Qwen3** |
|
||
| GQA (KV heads) | 14 full | 16 full | **8 (16Q/8KV)** |
|
||
| QK-Norm | ❌ | ❌ | **✅** |
|
||
| rope_theta | 10k | 10k | **1M** |
|
||
|
||
---
|
||
|
||
## 📊 Evaluation
|
||
|
||
| Category | Hits | Score |
|
||
|---|---|---|
|
||
| Knowledge | 0/5 | 🔴 0% |
|
||
| Reasoning | 0/4 | 🔴 0% |
|
||
| Hallucination | 0/4 | 🔴 0% |
|
||
| Instruction | 2/4 | 🟡 50% |
|
||
| Coherence | 1/3 | 🔴 33% |
|
||
| **Overall** | — | **🔴 17%** |
|
||
|
||
> **Hallucination resistance** tests whether the model correctly declines or hedges
|
||
> on unanswerable questions (future events, fictional entities, impossible premises).
|
||
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
## 🍳 Training
|
||
|
||
### What's new in v5.1
|
||
|
||
| Change | v4.5 | v5.1 | Why |
|
||
|---|---|---|---|
|
||
| Sequence packing | ❌ padding waste | ✅ 100% tokens | ~3× more signal per step |
|
||
| Dataset quality | mixed web+instruction | GPT-4 quality-tiered | Faster loss reduction |
|
||
| Parameters | ~256 M | **~1218.3 M** (~1.218 B) | Better capacity |
|
||
| Datasets | 15 | **21** | More diversity |
|
||
| LR | 1e-4 | **2e-4** | 1e-4 was too conservative |
|
||
|
||
### Settings
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Hardware | GTX 1080 8 GB · Pascal · CUDA 6.1 |
|
||
| Precision | fp32 weights / fp16 AMP (GradScaler) |
|
||
| Optimizer | StovetopCooker (HyperNix, pre-Volta) + cosine |
|
||
| LR | `0.0002` cosine |
|
||
| Warmup | 8% |
|
||
| Embedding freeze | First 20% of steps |
|
||
| Effective batch | 8 × 512 = 4,096 tokens/step |
|
||
| Loss boost | ≤3 rounds of 75 steps if loss > 4.95 |
|
||
| Sequence packing | ✅ streaming, 1× epochs, 150,000 chunks cap |
|
||
| Grad clipping | 5.0 |
|
||
| Grad checkpointing | ✅ |
|
||
| Peak VRAM | 5.44 GB |
|
||
| Final loss | `2.0444` |
|
||
|
||
### Dataset Mix (21 datasets, quality-tiered)
|
||
|
||
| Tier | Dataset | Samples | Weight | Category |
|
||
|---|---|---|---|---|
|
||
| 1 | `Open-Orca/OpenOrca` | 40 k | 3.0× | GPT-4 reasoning |
|
||
| 1 | `meta-math/MetaMathQA` | 30 k | 2.8× | Math augmentation |
|
||
| 1 | `Roman1111111/claude-opus-4.6-10000x` | 10 k | 2.5× | Claude conversations |
|
||
| 1 | `WizardLM/WizardLM_evol_instruct_V2_196k` | 25 k | 2.5× | Evolved instruction |
|
||
| 1 | `WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K` | 25 k | 2.5× | Reasoning traces |
|
||
| 2 | `microsoft/orca-math-word-problems-200k` | 20 k | 2.2× | Math word problems |
|
||
| 2 | `lighteval/MATH-Hard` | 10 k | 2.2× | Hard math |
|
||
| 2 | `HuggingFaceH4/MATH-500` | 500 | 2.2× | Competition math |
|
||
| 2 | `garage-bAInd/Open-Platypus` | 25 k | 2.0× | Reasoning instruction |
|
||
| 2 | `teknium/OpenHermes-2.5` | 30 k | 2.0× | GPT-4 instruction |
|
||
| 3 | `ise-uiuc/Magicoder-OSS-Instruct-75K` | 20 k | 1.8× | Code instruction |
|
||
| 3 | `m-a-p/CodeFeedback-Filtered-Instruction` | 15 k | 1.8× | Code + feedback |
|
||
| 3 | `iamtarun/python_code_instructions_18k_alpaca` | 8 k | 1.6× | Python code |
|
||
| 3 | `nvidia/OpenCodeInstruct` | 20 k | 1.5× | Code instruction |
|
||
| 3 | `b-mc2/sql-create-context` | 6 k | 1.4× | SQL generation |
|
||
| 4 | `HuggingFaceH4/ultrachat_200k` | 30 k | 1.5× | Multi-turn chat |
|
||
| 4 | `databricks/databricks-dolly-15k` | 15 k | 1.2× | Instruction following |
|
||
| 4 | `Amod/mental_health_counseling_conversations` | 5 k | 1.0× | Counseling chat |
|
||
| 4 | `mlabonne/guanaco-llama2-1k` | 1 k | 1.0× | General QA |
|
||
| 5 | `ray0rf1re/FineWeb-Nano` | 20 k | 0.8× | Web text |
|
||
| 5 | `ray0rf1re/hyper-pip` | 85 | 3.0× | HyperNix pip data |
|
||
| 3 | `flytech/python-codes-25k` | 20 k | 1.7× | Python code solutions |
|
||
| 3 | `ByteDance-Seed/Code-Contests-Plus` | 15 k | 1.6× | Competitive coding |
|
||
| 1 | `open-thoughts/OpenThoughts-TB-dev` | 20 k | 2.3× | Verified thinking traces |
|
||
| 6 | `Nix-ai/cat-math-v1` | 5 k | 0.3× | Cat math (niche) |
|
||
| 6 | `Nix-ai/Cat-v2.8XXXL-plus` | 5 k | 0.3× | Cat general (niche) |
|
||
|
||
---
|
||
|
||
## 🚀 Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"ray0rf1re/Nano-Nano_v5.1", torch_dtype="auto", device_map="auto")
|
||
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-Nano_v5.1")
|
||
|
||
def chat(prompt: str, max_new_tokens: int = 256) -> str:
|
||
# <think> opens the reasoning block; model outputs reasoning then </think> then answer
|
||
text = ("<|im_start|>user
|
||
" + prompt + "<|im_end|>
|
||
"
|
||
"<|im_start|>assistant
|
||
<think>
|
||
")
|
||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||
out = model.generate(
|
||
**inputs, max_new_tokens=max_new_tokens,
|
||
do_sample=True, temperature=0.7, top_p=0.9,
|
||
repetition_penalty=1.1,
|
||
pad_token_id=tokenizer.eos_token_id,
|
||
)
|
||
return tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:],
|
||
skip_special_tokens=True).strip()
|
||
|
||
print(chat("Write a Python function to merge two sorted lists."))
|
||
print(chat("Solve: if 3x + 7 = 22, what is x?"))
|
||
print(chat("Explain transformer attention in simple terms."))
|
||
```
|
||
|
||
---
|
||
|
||
## ⚠️ Limitations
|
||
|
||
- Context limited to **2 048 tokens**
|
||
- Trained on **0.01 B tokens** — far below production scale
|
||
- Pascal GPU (GTX 1080): fp16 AMP only, no bf16
|
||
- Not RLHF/DPO aligned
|
||
|
||
---
|
||
|
||
## 📜 Citation
|
||
|
||
```bibtex
|
||
@misc{nano-nano-v5,
|
||
author = {ray0rf1re},
|
||
title = {Nano-Nano v5.1: 300M LLaMA with Sequence Packing},
|
||
year = {2026},
|
||
publisher = {HuggingFace},
|
||
howpublished = {https://huggingface.co/ray0rf1re/Nano-Nano_v5.1},
|
||
}
|
||
``` |