初始化项目，由ModelHub XC社区提供模型

Model: ray0rf1re/Nano-nano_v4.5 Source: Original Platform
2026-07-17 09:41:11 +08:00
commit 12583ebf3b
13 changed files with 250988 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,245 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- llama
+- causal-lm
+- text-generation
+- instruction-tuned
+- nano
+- v4.5
+- pytorch
+pipeline_tag: text-generation
+datasets:
+- Roman1111111/claude-opus-4.6-10000x
+- WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K
+- HuggingFaceH4/MATH-500
+- lighteval/MATH-Hard
+- garage-bAInd/Open-Platypus
+- nvidia/OpenCodeInstruct
+- iamtarun/python_code_instructions_18k_alpaca
+- b-mc2/sql-create-context
+- teknium/OpenHermes-2.5
+- ray0rf1re/FineWeb-Nano
+- tonytins/chat-dataset
+- Amod/mental_health_counseling_conversations
+- databricks/databricks-dolly-15k
+- mlabonne/guanaco-llama2-1k
+- fka/awesome-chatgpt-prompts
+- ray0rf1re/hyper-pip
+- HuggingFaceH4/ultrachat_200k
+model-index:
+- name: Nano-nano v4.5
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    metrics:
+    - name: Training Loss
+      type: loss
+      value: 5.1763
+    - name: Overall Eval Score
+      type: accuracy
+      value: 0.1667
+    - name: Knowledge
+      type: accuracy
+      value: 0
+    - name: Reasoning
+      type: accuracy
+      value: 0
+    - name: Hallucination Resistance
+      type: accuracy
+      value: 0
+    - name: Instruction Following
+      type: accuracy
+      value: 0.5
+    - name: Coherence
+      type: accuracy
+      value: 0.3333
+new_version: ray0rf1re/Nano-nano-4.6
+---
+
+<div align="center">
+
+# 🧠 Nano-nano v4.5
+
+### ~255.7 M · LLaMA · Instruction-tuned · From scratch
+
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Loss](https://img.shields.io/badge/Train_Loss-5.176-orange)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
+[![Eval](https://img.shields.io/badge/Eval_Score-16%25-orange)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
+[![Datasets](https://img.shields.io/badge/Datasets-15-blueviolet)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
+
+Successor to [Nano-nano v4](ray0rf1re/Nano-nano-v4).  
+Same architecture family, **~8.5% larger**, trained from scratch on 15 carefully weighted datasets.
+
+</div>
+
+---
+
+## 📋 Quick Facts
+
+| | |
+|---|---|
+| Architecture | LLaMA (decoder-only) |
+| Parameters | ~255.7 M |
+| Context length | 2 048 tokens |
+| Vocabulary | 50,264 tokens |
+| Training loss | `5.1763` |
+| Eval score | `16.7%` |
+| Trained on | 0.08 B tokens |
+| Hardware | NVIDIA GTX 1080 8 GB (Pascal) |
+| Trained | 2026-05-09 22:50 |
+
+---
+
+## 🏗️ Architecture
+
+Standard LLaMA decoder-only transformer. Scaled **~8.5% wider + 1 extra layer** vs v4.
+
+| Hyperparameter | v4 | **v4.5** |
+|---|---|---|
+| Parameters | ~236 M | **~255.7 M** |
+| `hidden_size` | 896 | 896 |
+| `intermediate_size` | 2 688 | **2 912** |
+| `num_hidden_layers` | 14 | **15** |
+| `num_attention_heads` | 14 | 14 |
+| `num_key_value_heads` | 14 | 14 |
+| `head_dim` | 64 | 64 |
+| `vocab_size` | 50 264 | 50,264 |
+| `max_position_embeddings` | 1 024 | **2 048** |
+| `rms_norm_eps` | 1e-6 | 1e-6 |
+| `rope_theta` | 10 000 | 10 000 |
+| `hidden_act` | SiLU | SiLU |
+| `tie_word_embeddings` | False | False |
+| `attention_bias` | False | False |
+| `mlp_bias` | False | False |
+
+---
+
+## 📊 Evaluation
+
+Automatically evaluated after training across 5 capability dimensions.
+
+| Category | Hits | Score |
+|---|---|---|
+| Knowledge | 0/5 | 🔴 0% |
+| Reasoning | 0/4 | 🔴 0% |
+| Hallucination | 0/4 | 🔴 0% |
+| Instruction | 2/4 | 🟡 50% |
+| Coherence | 1/3 | 🔴 33% |
+| **Overall** | — | **🔴 17%** |
+
+> **Hallucination resistance** — whether the model appropriately declines questions
+> about future events, fictional entities, or impossible premises rather than confabulating.
+
+![Category Scores](eval_category_scores.png)
+![Hallucination](eval_hallucination.png)
+![Training Curves](training_curves.png)
+
+---
+
+## 🍳 Training
+
+| Setting | Value |
+|---|---|
+| Hardware | GTX 1080 8 GB · Pascal · CUDA 6.1 |
+| Precision | fp32 weights / fp16 AMP (GradScaler) |
+| Optimizer | StovetopCooker (HyperNix, pre-Volta) |
+| LR | `0.0001` cosine decay |
+| Warmup | 6% of steps |
+| Embedding freeze | First 15% of steps |
+| Effective batch | 8 × 2048 = 16,384 tokens/step |
+| Steps | 5092 |
+| Total tokens | 0.08 B |
+| Grad clipping | 1.0 |
+| Grad checkpointing | ✅ |
+| Peak VRAM | 5.34 GB |
+| HyperNix | ✅ `freezer` · `StovetopCooker` · `old_fridge` · `new_fridge` · `smoke_alarm` · `pans` · `smoker` |
+
+### Dataset Mix
+
+| Dataset | Samples | Weight | Category |
+|---|---|---|---|
+| `Roman1111111/claude-opus-4.6-10000x` | 10 k | 2.5× | Claude conversations |
+| `WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K` | 25 k | 2.0× | Reasoning / thinking |
+| `HuggingFaceH4/MATH-500` | 500 | 2.0× | Competition math |
+| `lighteval/MATH-Hard` | 10 k | 2.0× | Hard math |
+| `garage-bAInd/Open-Platypus` | 25 k | 1.8× | Reasoning instruction |
+| `iamtarun/python_code_instructions_18k_alpaca` | 8 k | 1.6× | Python code |
+| `b-mc2/sql-create-context` | 6 k | 1.4× | SQL code |
+| `nvidia/OpenCodeInstruct` | 30 k | 1.5× | Code instruction |
+| `teknium/OpenHermes-2.5` | 30 k | 1.5× | General instruction |
+| `Amod/mental_health_counseling_conversations` | 5 k | 1.2× | Chat / counseling |
+| `ray0rf1re/FineWeb-Nano` | 50 k | 1.0× | Web text |
+| `tonytins/chat-dataset` | 10 k | 1.0× | Conversation |
+| `databricks/databricks-dolly-15k` | 15 k | 1.0× | Instruction following |
+| `mlabonne/guanaco-llama2-1k` | 1 k | 1.0× | General QA |
+| `ray0rf1re/hyper-pip` | 20 k | 2.0× | HyperNix pip data |
+| `HuggingFaceH4/ultrachat_200k` | 30 k | 1.5× | Multi-turn chat |
+| `fka/awesome-chatgpt-prompts` | 5 k | 0.8× | Prompt engineering |
+
+---
+
+## 🚀 Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "ray0rf1re/Nano-nano_v4.5",
+    torch_dtype="auto",
+    device_map="auto",
+)
+tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-nano_v4.5")
+
+def generate(prompt: str, max_new_tokens: int = 256) -> str:
+    text   = f"### Instruction:
+{prompt}
+
+### Response:
+"
+    inputs = tokenizer(text, return_tensors="pt").to(model.device)
+    out    = model.generate(
+        **inputs,
+        max_new_tokens  = max_new_tokens,
+        do_sample       = True,
+        temperature     = 0.7,
+        top_p           = 0.9,
+        repetition_penalty = 1.1,
+        pad_token_id    = tokenizer.eos_token_id,
+    )
+    new_ids = out[0][inputs["input_ids"].shape[-1]:]
+    return tokenizer.decode(new_ids, skip_special_tokens=True).strip()
+
+# Examples
+print(generate("Write a Python function to reverse a linked list."))
+print(generate("What is the capital of France?"))
+print(generate("Explain gradient descent in simple terms."))
+```
+
+---
+
+## ⚠️ Limitations
+
+- Context limited to **1 024 tokens** — unsuitable for long documents
+- Trained on **0.08 B tokens** — far less than production models
+- May hallucinate on obscure or out-of-distribution queries
+- Not RLHF/DPO aligned — outputs may vary in safety and tone
+- Pascal GPU constraint (GTX 1080): fp32/fp16 only, no bf16
+
+---
+
+## 📜 Citation
+
+```bibtex
+@misc{nano-nano-v45,
+  author       = {ray0rf1re},
+  title        = {Nano-nano v4.5: Compact LLaMA-Family Causal LM},
+  year         = {2026},
+  publisher    = {HuggingFace},
+  howpublished = {https://huggingface.co/ray0rf1re/Nano-nano_v4.5},
+}
+```