Files

ModelHub XC 2f5adbaa8b 初始化项目，由ModelHub XC社区提供模型

Model: ray0rf1re/Nano-Nano_v5.1
Source: Original Platform

2026-07-15 21:56:43 +08:00

9.1 KiB

Raw Permalink Blame History

language, license, library_name, tags, pipeline_tag, datasets, model-index

language

license

library_name

🧠 Nano-Nano v5.1

~1218.3 M · Qwen3 · 300M · GQA + QK-Norm · Sequence-Packed · 26 Datasets

Fully redesigned successor to Nano-nano v4.5.
~298M Qwen3 parameters trained with sequence packing on a quality-tiered 34-dataset mix. Features loss-boost system: auto-extends training if loss > 4.95 (up to 3×75 steps).
Goal: loss < 2.5 through compute efficiency, not raw scale.

📋 Summary


Architecture	LLaMA decoder-only
Parameters	~1218.3 M
Context	2 048 tokens
Vocabulary	50,304 tokens
Training loss	`2.0444`
Eval score	`16.7%`
Tokens trained	0.01 B (sequence-packed)
Hardware	GTX 1080 8 GB (Pascal)

🏗️ Architecture (v4 → v4.5 → v5.1)

Hyperparameter	v4	v4.5	v5.1
Parameters	~236 M	~256 M	~1218.3 M (~1.218 B)
`hidden_size`	896	896	1 024
`intermediate_size`	2 688	2 912	2 730 (8/3×hidden)
`num_hidden_layers`	14	15	16
`num_attention_heads`	14	14	16
`num_key_value_heads`	14	14	16
`head_dim`	64	64	64
`vocab_size`	50 264	50 264	50,304
`max_position_embeddings`	1 024	2 048	2 048
`rms_norm_eps`	1e-6	1e-6	1e-5
`rope_theta`	10 000	10 000	10 000
`rope_scaling`	—	linear 2×	linear 2×
`tie_word_embeddings`	False	False	False
Sequence packing	❌	❌	✅ 1× packed
Architecture	LLaMA	LLaMA	Qwen3
GQA (KV heads)	14 full	16 full	8 (16Q/8KV)
QK-Norm	❌	❌	✅
rope_theta	10k	10k	1M

📊 Evaluation

Category	Hits	Score
Knowledge	0/5	🔴 0%
Reasoning	0/4	🔴 0%
Hallucination	0/4	🔴 0%
Instruction	2/4	🟡 50%
Coherence	1/3	🔴 33%
Overall	—	🔴 17%

Hallucination resistance tests whether the model correctly declines or hedges on unanswerable questions (future events, fictional entities, impossible premises).

🍳 Training

What's new in v5.1

Change	v4.5	v5.1	Why
Sequence packing	❌ padding waste	✅ 100% tokens	~3× more signal per step
Dataset quality	mixed web+instruction	GPT-4 quality-tiered	Faster loss reduction
Parameters	~256 M	~1218.3 M (~1.218 B)	Better capacity
Datasets	15	21	More diversity
LR	1e-4	2e-4	1e-4 was too conservative

Settings

Setting	Value
Hardware	GTX 1080 8 GB · Pascal · CUDA 6.1
Precision	fp32 weights / fp16 AMP (GradScaler)
Optimizer	StovetopCooker (HyperNix, pre-Volta) + cosine
LR	`0.0002` cosine
Warmup	8%
Embedding freeze	First 20% of steps
Effective batch	8 × 512 = 4,096 tokens/step
Loss boost	≤3 rounds of 75 steps if loss > 4.95
Sequence packing	✅ streaming, 1× epochs, 150,000 chunks cap
Grad clipping	5.0
Grad checkpointing	✅
Peak VRAM	5.44 GB
Final loss	`2.0444`

Dataset Mix (21 datasets, quality-tiered)

Tier	Dataset	Samples	Weight	Category
1	`Open-Orca/OpenOrca`	40 k	3.0×	GPT-4 reasoning
1	`meta-math/MetaMathQA`	30 k	2.8×	Math augmentation
1	`Roman1111111/claude-opus-4.6-10000x`	10 k	2.5×	Claude conversations
1	`WizardLM/WizardLM_evol_instruct_V2_196k`	25 k	2.5×	Evolved instruction
1	`WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K`	25 k	2.5×	Reasoning traces
2	`microsoft/orca-math-word-problems-200k`	20 k	2.2×	Math word problems
2	`lighteval/MATH-Hard`	10 k	2.2×	Hard math
2	`HuggingFaceH4/MATH-500`	500	2.2×	Competition math
2	`garage-bAInd/Open-Platypus`	25 k	2.0×	Reasoning instruction
2	`teknium/OpenHermes-2.5`	30 k	2.0×	GPT-4 instruction
3	`ise-uiuc/Magicoder-OSS-Instruct-75K`	20 k	1.8×	Code instruction
3	`m-a-p/CodeFeedback-Filtered-Instruction`	15 k	1.8×	Code + feedback
3	`iamtarun/python_code_instructions_18k_alpaca`	8 k	1.6×	Python code
3	`nvidia/OpenCodeInstruct`	20 k	1.5×	Code instruction
3	`b-mc2/sql-create-context`	6 k	1.4×	SQL generation
4	`HuggingFaceH4/ultrachat_200k`	30 k	1.5×	Multi-turn chat
4	`databricks/databricks-dolly-15k`	15 k	1.2×	Instruction following
4	`Amod/mental_health_counseling_conversations`	5 k	1.0×	Counseling chat
4	`mlabonne/guanaco-llama2-1k`	1 k	1.0×	General QA
5	`ray0rf1re/FineWeb-Nano`	20 k	0.8×	Web text
5	`ray0rf1re/hyper-pip`	85	3.0×	HyperNix pip data
3	`flytech/python-codes-25k`	20 k	1.7×	Python code solutions
3	`ByteDance-Seed/Code-Contests-Plus`	15 k	1.6×	Competitive coding
1	`open-thoughts/OpenThoughts-TB-dev`	20 k	2.3×	Verified thinking traces
6	`Nix-ai/cat-math-v1`	5 k	0.3×	Cat math (niche)
6	`Nix-ai/Cat-v2.8XXXL-plus`	5 k	0.3×	Cat general (niche)

🚀 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ray0rf1re/Nano-Nano_v5.1", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-Nano_v5.1")

def chat(prompt: str, max_new_tokens: int = 256) -> str:
    # <think> opens the reasoning block; model outputs reasoning then </think> then answer
    text = ("<|im_start|>user
" + prompt + "<|im_end|>
"
            "<|im_start|>assistant
<think>
")
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(
        **inputs, max_new_tokens=max_new_tokens,
        do_sample=True, temperature=0.7, top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    return tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:],
                            skip_special_tokens=True).strip()

print(chat("Write a Python function to merge two sorted lists."))
print(chat("Solve: if 3x + 7 = 22, what is x?"))
print(chat("Explain transformer attention in simple terms."))

⚠️ Limitations

Context limited to 2 048 tokens
Trained on 0.01 B tokens — far below production scale
Pascal GPU (GTX 1080): fp16 AMP only, no bf16
Not RLHF/DPO aligned

📜 Citation

@misc{nano-nano-v5,
  author       = {ray0rf1re},
  title        = {Nano-Nano v5.1: 300M LLaMA with Sequence Packing},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {https://huggingface.co/ray0rf1re/Nano-Nano_v5.1},
}

9.1 KiB Raw Permalink Blame History Unescape Escape