Nano-Nano_v5.1/README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- llama
- causal-lm
- text-generation
- instruction-tuned
- nano
- v5.1
- pytorch
- sequence-packing
pipeline_tag: text-generation
datasets:
- Open-Orca/OpenOrca
- meta-math/MetaMathQA
- Roman1111111/claude-opus-4.6-10000x
- WizardLM/WizardLM_evol_instruct_V2_196k
- WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K
- microsoft/orca-math-word-problems-200k
- lighteval/MATH-Hard
- HuggingFaceH4/MATH-500
- garage-bAInd/Open-Platypus
- teknium/OpenHermes-2.5
- ise-uiuc/Magicoder-OSS-Instruct-75K
- m-a-p/CodeFeedback-Filtered-Instruction
- iamtarun/python_code_instructions_18k_alpaca
- nvidia/OpenCodeInstruct
- b-mc2/sql-create-context
- HuggingFaceH4/ultrachat_200k
- databricks/databricks-dolly-15k
- Amod/mental_health_counseling_conversations
- mlabonne/guanaco-llama2-1k
- ray0rf1re/FineWeb-Nano
- ray0rf1re/hyper-pip
- flytech/python-codes-25k
- ByteDance-Seed/Code-Contests-Plus
- open-thoughts/OpenThoughts-TB-dev
- Nix-ai/cat-math-v1
- Nix-ai/Cat-v2.8XXXL-plus
- ray0rf1re/claude1255x9
- ray0rf1re/archlinux-v1
- HuggingFaceFW/fineweb-edu
- ajibawa-2023/Code-74k-ShareGPT
- codeparrot/apps
- allenai/moral_stories
- mrm8488/shell-commands-prompts
model-index:
- name: Nano-Nano v5.1
  results:
  - task:
      type: text-generation
      name: Text Generation
    metrics:
    - name: Training Loss
      type: loss
      value: 2.0444
    - name: Overall Eval Score
      type: accuracy
      value: 0.1667
    - name: Knowledge
      type: accuracy
      value: 0
    - name: Reasoning
      type: accuracy
      value: 0
    - name: Hallucination Resistance
      type: accuracy
      value: 0
    - name: Instruction Following
      type: accuracy
      value: 0.5
    - name: Coherence
      type: accuracy
      value: 0.3333
---

<div align="center">

# 🧠 Nano-Nano v5.1

### ~1218.3 M · Qwen3 · 300M · GQA + QK-Norm · Sequence-Packed · 26 Datasets

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Loss](https://img.shields.io/badge/Train_Loss-2.044-brightgreen)](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
[![Eval](https://img.shields.io/badge/Eval_Score-16%25-orange)](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
[![Params](https://img.shields.io/badge/Parameters-~1.218B-blue)](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)
[![Datasets](https://img.shields.io/badge/Datasets-34-blueviolet)](https://huggingface.co/ray0rf1re/Nano-Nano_v5.1)

**Fully redesigned** successor to Nano-nano v4.5.  
~298M Qwen3 parameters trained with **sequence packing** on a quality-tiered 34-dataset mix.
Features loss-boost system: auto-extends training if loss > 4.95 (up to 3×75 steps).  
Goal: loss < 2.5 through compute efficiency, not raw scale.

</div>

---

## 📋 Summary

| | |
|---|---|
| Architecture | LLaMA decoder-only |
| Parameters | ~1218.3 M |
| Context | 2 048 tokens |
| Vocabulary | 50,304 tokens |
| Training loss | `2.0444` |
| Eval score | `16.7%` |
| Tokens trained | 0.01 B (sequence-packed) |
| Hardware | GTX 1080 8 GB (Pascal) |

---

## 🏗️ Architecture (v4 → v4.5 → v5.1)

| Hyperparameter | v4 | v4.5 | **v5.1** |
|---|---|---|---|
| Parameters | ~236 M | ~256 M | **~1218.3 M** (~1.218 B) |
| `hidden_size` | 896 | 896 | **1 024** |
| `intermediate_size` | 2 688 | 2 912 | **2 730** (8/3×hidden) |
| `num_hidden_layers` | 14 | 15 | **16** |
| `num_attention_heads` | 14 | 14 | **16** |
| `num_key_value_heads` | 14 | 14 | **16** |
| `head_dim` | 64 | 64 | 64 |
| `vocab_size` | 50 264 | 50 264 | 50,304 |
| `max_position_embeddings` | 1 024 | 2 048 | 2 048 |
| `rms_norm_eps` | 1e-6 | 1e-6 | **1e-5** |
| `rope_theta` | 10 000 | 10 000 | 10 000 |
| `rope_scaling` | — | linear 2× | linear 2× |
| `tie_word_embeddings` | False | False | False |
| Sequence packing | ❌ | ❌ | **✅ 1× packed** |
| Architecture | LLaMA | LLaMA | **Qwen3** |
| GQA (KV heads) | 14 full | 16 full | **8 (16Q/8KV)** |
| QK-Norm | ❌ | ❌ | **✅** |
| rope_theta | 10k | 10k | **1M** |

---

## 📊 Evaluation

| Category | Hits | Score |
|---|---|---|
| Knowledge | 0/5 | 🔴 0% |
| Reasoning | 0/4 | 🔴 0% |
| Hallucination | 0/4 | 🔴 0% |
| Instruction | 2/4 | 🟡 50% |
| Coherence | 1/3 | 🔴 33% |
| **Overall** | — | **🔴 17%** |

> **Hallucination resistance** tests whether the model correctly declines or hedges
> on unanswerable questions (future events, fictional entities, impossible premises).

![Category Scores](eval_category_scores.png)
![Hallucination Resistance](eval_hallucination.png)
![Training Curves](training_curves.png)

---

## 🍳 Training

### What's new in v5.1

| Change | v4.5 | v5.1 | Why |
|---|---|---|---|
| Sequence packing | ❌ padding waste | ✅ 100% tokens | ~3× more signal per step |
| Dataset quality | mixed web+instruction | GPT-4 quality-tiered | Faster loss reduction |
| Parameters | ~256 M | **~1218.3 M** (~1.218 B) | Better capacity |
| Datasets | 15 | **21** | More diversity |
| LR | 1e-4 | **2e-4** | 1e-4 was too conservative |

### Settings

| Setting | Value |
|---|---|
| Hardware | GTX 1080 8 GB · Pascal · CUDA 6.1 |
| Precision | fp32 weights / fp16 AMP (GradScaler) |
| Optimizer | StovetopCooker (HyperNix, pre-Volta) + cosine |
| LR | `0.0002` cosine |
| Warmup | 8% |
| Embedding freeze | First 20% of steps |
| Effective batch | 8 × 512 = 4,096 tokens/step |
| Loss boost | ≤3 rounds of 75 steps if loss > 4.95 |
| Sequence packing | ✅ streaming, 1× epochs, 150,000 chunks cap |
| Grad clipping | 5.0 |
| Grad checkpointing | ✅ |
| Peak VRAM | 5.44 GB |
| Final loss | `2.0444` |

### Dataset Mix (21 datasets, quality-tiered)

| Tier | Dataset | Samples | Weight | Category |
|---|---|---|---|---|
| 1 | `Open-Orca/OpenOrca` | 40 k | 3.0× | GPT-4 reasoning |
| 1 | `meta-math/MetaMathQA` | 30 k | 2.8× | Math augmentation |
| 1 | `Roman1111111/claude-opus-4.6-10000x` | 10 k | 2.5× | Claude conversations |
| 1 | `WizardLM/WizardLM_evol_instruct_V2_196k` | 25 k | 2.5× | Evolved instruction |
| 1 | `WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K` | 25 k | 2.5× | Reasoning traces |
| 2 | `microsoft/orca-math-word-problems-200k` | 20 k | 2.2× | Math word problems |
| 2 | `lighteval/MATH-Hard` | 10 k | 2.2× | Hard math |
| 2 | `HuggingFaceH4/MATH-500` | 500 | 2.2× | Competition math |
| 2 | `garage-bAInd/Open-Platypus` | 25 k | 2.0× | Reasoning instruction |
| 2 | `teknium/OpenHermes-2.5` | 30 k | 2.0× | GPT-4 instruction |
| 3 | `ise-uiuc/Magicoder-OSS-Instruct-75K` | 20 k | 1.8× | Code instruction |
| 3 | `m-a-p/CodeFeedback-Filtered-Instruction` | 15 k | 1.8× | Code + feedback |
| 3 | `iamtarun/python_code_instructions_18k_alpaca` | 8 k | 1.6× | Python code |
| 3 | `nvidia/OpenCodeInstruct` | 20 k | 1.5× | Code instruction |
| 3 | `b-mc2/sql-create-context` | 6 k | 1.4× | SQL generation |
| 4 | `HuggingFaceH4/ultrachat_200k` | 30 k | 1.5× | Multi-turn chat |
| 4 | `databricks/databricks-dolly-15k` | 15 k | 1.2× | Instruction following |
| 4 | `Amod/mental_health_counseling_conversations` | 5 k | 1.0× | Counseling chat |
| 4 | `mlabonne/guanaco-llama2-1k` | 1 k | 1.0× | General QA |
| 5 | `ray0rf1re/FineWeb-Nano` | 20 k | 0.8× | Web text |
| 5 | `ray0rf1re/hyper-pip` | 85 | 3.0× | HyperNix pip data |
| 3 | `flytech/python-codes-25k` | 20 k | 1.7× | Python code solutions |
| 3 | `ByteDance-Seed/Code-Contests-Plus` | 15 k | 1.6× | Competitive coding |
| 1 | `open-thoughts/OpenThoughts-TB-dev` | 20 k | 2.3× | Verified thinking traces |
| 6 | `Nix-ai/cat-math-v1` | 5 k | 0.3× | Cat math (niche) |
| 6 | `Nix-ai/Cat-v2.8XXXL-plus` | 5 k | 0.3× | Cat general (niche) |

---

## 🚀 Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ray0rf1re/Nano-Nano_v5.1", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-Nano_v5.1")

def chat(prompt: str, max_new_tokens: int = 256) -> str:
    # <think> opens the reasoning block; model outputs reasoning then </think> then answer
    text = ("<|im_start|>user
" + prompt + "<|im_end|>
"
            "<|im_start|>assistant
<think>
")
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(
        **inputs, max_new_tokens=max_new_tokens,
        do_sample=True, temperature=0.7, top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    return tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:],
                            skip_special_tokens=True).strip()

print(chat("Write a Python function to merge two sorted lists."))
print(chat("Solve: if 3x + 7 = 22, what is x?"))
print(chat("Explain transformer attention in simple terms."))
```

---

## ⚠️ Limitations

- Context limited to **2 048 tokens**
- Trained on **0.01 B tokens** — far below production scale
- Pascal GPU (GTX 1080): fp16 AMP only, no bf16
- Not RLHF/DPO aligned

---

## 📜 Citation

```bibtex
@misc{nano-nano-v5,
  author       = {ray0rf1re},
  title        = {Nano-Nano v5.1: 300M LLaMA with Sequence Packing},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {https://huggingface.co/ray0rf1re/Nano-Nano_v5.1},
}
```