Nano-nano_v4.5/README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- llama
- causal-lm
- text-generation
- instruction-tuned
- nano
- v4.5
- pytorch
pipeline_tag: text-generation
datasets:
- Roman1111111/claude-opus-4.6-10000x
- WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K
- HuggingFaceH4/MATH-500
- lighteval/MATH-Hard
- garage-bAInd/Open-Platypus
- nvidia/OpenCodeInstruct
- iamtarun/python_code_instructions_18k_alpaca
- b-mc2/sql-create-context
- teknium/OpenHermes-2.5
- ray0rf1re/FineWeb-Nano
- tonytins/chat-dataset
- Amod/mental_health_counseling_conversations
- databricks/databricks-dolly-15k
- mlabonne/guanaco-llama2-1k
- fka/awesome-chatgpt-prompts
- ray0rf1re/hyper-pip
- HuggingFaceH4/ultrachat_200k
model-index:
- name: Nano-nano v4.5
  results:
  - task:
      type: text-generation
      name: Text Generation
    metrics:
    - name: Training Loss
      type: loss
      value: 5.1763
    - name: Overall Eval Score
      type: accuracy
      value: 0.1667
    - name: Knowledge
      type: accuracy
      value: 0
    - name: Reasoning
      type: accuracy
      value: 0
    - name: Hallucination Resistance
      type: accuracy
      value: 0
    - name: Instruction Following
      type: accuracy
      value: 0.5
    - name: Coherence
      type: accuracy
      value: 0.3333
new_version: ray0rf1re/Nano-nano-4.6
---

<div align="center">

# 🧠 Nano-nano v4.5

### ~255.7 M · LLaMA · Instruction-tuned · From scratch

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Loss](https://img.shields.io/badge/Train_Loss-5.176-orange)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
[![Eval](https://img.shields.io/badge/Eval_Score-16%25-orange)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
[![Datasets](https://img.shields.io/badge/Datasets-15-blueviolet)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)

Successor to [Nano-nano v4](ray0rf1re/Nano-nano-v4).  
Same architecture family, **~8.5% larger**, trained from scratch on 15 carefully weighted datasets.

</div>

---

## 📋 Quick Facts

| | |
|---|---|
| Architecture | LLaMA (decoder-only) |
| Parameters | ~255.7 M |
| Context length | 2 048 tokens |
| Vocabulary | 50,264 tokens |
| Training loss | `5.1763` |
| Eval score | `16.7%` |
| Trained on | 0.08 B tokens |
| Hardware | NVIDIA GTX 1080 8 GB (Pascal) |
| Trained | 2026-05-09 22:50 |

---

## 🏗️ Architecture

Standard LLaMA decoder-only transformer. Scaled **~8.5% wider + 1 extra layer** vs v4.

| Hyperparameter | v4 | **v4.5** |
|---|---|---|
| Parameters | ~236 M | **~255.7 M** |
| `hidden_size` | 896 | 896 |
| `intermediate_size` | 2 688 | **2 912** |
| `num_hidden_layers` | 14 | **15** |
| `num_attention_heads` | 14 | 14 |
| `num_key_value_heads` | 14 | 14 |
| `head_dim` | 64 | 64 |
| `vocab_size` | 50 264 | 50,264 |
| `max_position_embeddings` | 1 024 | **2 048** |
| `rms_norm_eps` | 1e-6 | 1e-6 |
| `rope_theta` | 10 000 | 10 000 |
| `hidden_act` | SiLU | SiLU |
| `tie_word_embeddings` | False | False |
| `attention_bias` | False | False |
| `mlp_bias` | False | False |

---

## 📊 Evaluation

Automatically evaluated after training across 5 capability dimensions.

| Category | Hits | Score |
|---|---|---|
| Knowledge | 0/5 | 🔴 0% |
| Reasoning | 0/4 | 🔴 0% |
| Hallucination | 0/4 | 🔴 0% |
| Instruction | 2/4 | 🟡 50% |
| Coherence | 1/3 | 🔴 33% |
| **Overall** | — | **🔴 17%** |

> **Hallucination resistance** — whether the model appropriately declines questions
> about future events, fictional entities, or impossible premises rather than confabulating.

![Category Scores](eval_category_scores.png)
![Hallucination](eval_hallucination.png)
![Training Curves](training_curves.png)

---

## 🍳 Training

| Setting | Value |
|---|---|
| Hardware | GTX 1080 8 GB · Pascal · CUDA 6.1 |
| Precision | fp32 weights / fp16 AMP (GradScaler) |
| Optimizer | StovetopCooker (HyperNix, pre-Volta) |
| LR | `0.0001` cosine decay |
| Warmup | 6% of steps |
| Embedding freeze | First 15% of steps |
| Effective batch | 8 × 2048 = 16,384 tokens/step |
| Steps | 5092 |
| Total tokens | 0.08 B |
| Grad clipping | 1.0 |
| Grad checkpointing | ✅ |
| Peak VRAM | 5.34 GB |
| HyperNix | ✅ `freezer` · `StovetopCooker` · `old_fridge` · `new_fridge` · `smoke_alarm` · `pans` · `smoker` |

### Dataset Mix

| Dataset | Samples | Weight | Category |
|---|---|---|---|
| `Roman1111111/claude-opus-4.6-10000x` | 10 k | 2.5× | Claude conversations |
| `WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K` | 25 k | 2.0× | Reasoning / thinking |
| `HuggingFaceH4/MATH-500` | 500 | 2.0× | Competition math |
| `lighteval/MATH-Hard` | 10 k | 2.0× | Hard math |
| `garage-bAInd/Open-Platypus` | 25 k | 1.8× | Reasoning instruction |
| `iamtarun/python_code_instructions_18k_alpaca` | 8 k | 1.6× | Python code |
| `b-mc2/sql-create-context` | 6 k | 1.4× | SQL code |
| `nvidia/OpenCodeInstruct` | 30 k | 1.5× | Code instruction |
| `teknium/OpenHermes-2.5` | 30 k | 1.5× | General instruction |
| `Amod/mental_health_counseling_conversations` | 5 k | 1.2× | Chat / counseling |
| `ray0rf1re/FineWeb-Nano` | 50 k | 1.0× | Web text |
| `tonytins/chat-dataset` | 10 k | 1.0× | Conversation |
| `databricks/databricks-dolly-15k` | 15 k | 1.0× | Instruction following |
| `mlabonne/guanaco-llama2-1k` | 1 k | 1.0× | General QA |
| `ray0rf1re/hyper-pip` | 20 k | 2.0× | HyperNix pip data |
| `HuggingFaceH4/ultrachat_200k` | 30 k | 1.5× | Multi-turn chat |
| `fka/awesome-chatgpt-prompts` | 5 k | 0.8× | Prompt engineering |

---

## 🚀 Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ray0rf1re/Nano-nano_v4.5",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-nano_v4.5")

def generate(prompt: str, max_new_tokens: int = 256) -> str:
    text   = f"### Instruction:
{prompt}

### Response:
"
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    out    = model.generate(
        **inputs,
        max_new_tokens  = max_new_tokens,
        do_sample       = True,
        temperature     = 0.7,
        top_p           = 0.9,
        repetition_penalty = 1.1,
        pad_token_id    = tokenizer.eos_token_id,
    )
    new_ids = out[0][inputs["input_ids"].shape[-1]:]
    return tokenizer.decode(new_ids, skip_special_tokens=True).strip()

# Examples
print(generate("Write a Python function to reverse a linked list."))
print(generate("What is the capital of France?"))
print(generate("Explain gradient descent in simple terms."))
```

---

## ⚠️ Limitations

- Context limited to **1 024 tokens** — unsuitable for long documents
- Trained on **0.08 B tokens** — far less than production models
- May hallucinate on obscure or out-of-distribution queries
- Not RLHF/DPO aligned — outputs may vary in safety and tone
- Pascal GPU constraint (GTX 1080): fp32/fp16 only, no bf16

---

## 📜 Citation

```bibtex
@misc{nano-nano-v45,
  author       = {ray0rf1re},
  title        = {Nano-nano v4.5: Compact LLaMA-Family Causal LM},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {https://huggingface.co/ray0rf1re/Nano-nano_v4.5},
}
```