初始化项目,由ModelHub XC社区提供模型

Model: ray0rf1re/Nano-nano_v4.5
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-17 09:41:11 +08:00
commit 12583ebf3b
13 changed files with 250988 additions and 0 deletions

37
.gitattributes vendored Normal file
View File

@@ -0,0 +1,37 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
eval_category_scores.png filter=lfs diff=lfs merge=lfs -text
training_curves.png filter=lfs diff=lfs merge=lfs -text

245
README.md Normal file
View File

@@ -0,0 +1,245 @@
---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- llama
- causal-lm
- text-generation
- instruction-tuned
- nano
- v4.5
- pytorch
pipeline_tag: text-generation
datasets:
- Roman1111111/claude-opus-4.6-10000x
- WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K
- HuggingFaceH4/MATH-500
- lighteval/MATH-Hard
- garage-bAInd/Open-Platypus
- nvidia/OpenCodeInstruct
- iamtarun/python_code_instructions_18k_alpaca
- b-mc2/sql-create-context
- teknium/OpenHermes-2.5
- ray0rf1re/FineWeb-Nano
- tonytins/chat-dataset
- Amod/mental_health_counseling_conversations
- databricks/databricks-dolly-15k
- mlabonne/guanaco-llama2-1k
- fka/awesome-chatgpt-prompts
- ray0rf1re/hyper-pip
- HuggingFaceH4/ultrachat_200k
model-index:
- name: Nano-nano v4.5
results:
- task:
type: text-generation
name: Text Generation
metrics:
- name: Training Loss
type: loss
value: 5.1763
- name: Overall Eval Score
type: accuracy
value: 0.1667
- name: Knowledge
type: accuracy
value: 0
- name: Reasoning
type: accuracy
value: 0
- name: Hallucination Resistance
type: accuracy
value: 0
- name: Instruction Following
type: accuracy
value: 0.5
- name: Coherence
type: accuracy
value: 0.3333
new_version: ray0rf1re/Nano-nano-4.6
---
<div align="center">
# 🧠 Nano-nano v4.5
### ~255.7 M · LLaMA · Instruction-tuned · From scratch
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Loss](https://img.shields.io/badge/Train_Loss-5.176-orange)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
[![Eval](https://img.shields.io/badge/Eval_Score-16%25-orange)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
[![Datasets](https://img.shields.io/badge/Datasets-15-blueviolet)](https://huggingface.co/ray0rf1re/Nano-nano_v4.5)
Successor to [Nano-nano v4](ray0rf1re/Nano-nano-v4).
Same architecture family, **~8.5% larger**, trained from scratch on 15 carefully weighted datasets.
</div>
---
## 📋 Quick Facts
| | |
|---|---|
| Architecture | LLaMA (decoder-only) |
| Parameters | ~255.7 M |
| Context length | 2 048 tokens |
| Vocabulary | 50,264 tokens |
| Training loss | `5.1763` |
| Eval score | `16.7%` |
| Trained on | 0.08 B tokens |
| Hardware | NVIDIA GTX 1080 8 GB (Pascal) |
| Trained | 2026-05-09 22:50 |
---
## 🏗️ Architecture
Standard LLaMA decoder-only transformer. Scaled **~8.5% wider + 1 extra layer** vs v4.
| Hyperparameter | v4 | **v4.5** |
|---|---|---|
| Parameters | ~236 M | **~255.7 M** |
| `hidden_size` | 896 | 896 |
| `intermediate_size` | 2 688 | **2 912** |
| `num_hidden_layers` | 14 | **15** |
| `num_attention_heads` | 14 | 14 |
| `num_key_value_heads` | 14 | 14 |
| `head_dim` | 64 | 64 |
| `vocab_size` | 50 264 | 50,264 |
| `max_position_embeddings` | 1 024 | **2 048** |
| `rms_norm_eps` | 1e-6 | 1e-6 |
| `rope_theta` | 10 000 | 10 000 |
| `hidden_act` | SiLU | SiLU |
| `tie_word_embeddings` | False | False |
| `attention_bias` | False | False |
| `mlp_bias` | False | False |
---
## 📊 Evaluation
Automatically evaluated after training across 5 capability dimensions.
| Category | Hits | Score |
|---|---|---|
| Knowledge | 0/5 | 🔴 0% |
| Reasoning | 0/4 | 🔴 0% |
| Hallucination | 0/4 | 🔴 0% |
| Instruction | 2/4 | 🟡 50% |
| Coherence | 1/3 | 🔴 33% |
| **Overall** | — | **🔴 17%** |
> **Hallucination resistance** — whether the model appropriately declines questions
> about future events, fictional entities, or impossible premises rather than confabulating.
![Category Scores](eval_category_scores.png)
![Hallucination](eval_hallucination.png)
![Training Curves](training_curves.png)
---
## 🍳 Training
| Setting | Value |
|---|---|
| Hardware | GTX 1080 8 GB · Pascal · CUDA 6.1 |
| Precision | fp32 weights / fp16 AMP (GradScaler) |
| Optimizer | StovetopCooker (HyperNix, pre-Volta) |
| LR | `0.0001` cosine decay |
| Warmup | 6% of steps |
| Embedding freeze | First 15% of steps |
| Effective batch | 8 × 2048 = 16,384 tokens/step |
| Steps | 5092 |
| Total tokens | 0.08 B |
| Grad clipping | 1.0 |
| Grad checkpointing | ✅ |
| Peak VRAM | 5.34 GB |
| HyperNix | ✅ `freezer` · `StovetopCooker` · `old_fridge` · `new_fridge` · `smoke_alarm` · `pans` · `smoker` |
### Dataset Mix
| Dataset | Samples | Weight | Category |
|---|---|---|---|
| `Roman1111111/claude-opus-4.6-10000x` | 10 k | 2.5× | Claude conversations |
| `WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K` | 25 k | 2.0× | Reasoning / thinking |
| `HuggingFaceH4/MATH-500` | 500 | 2.0× | Competition math |
| `lighteval/MATH-Hard` | 10 k | 2.0× | Hard math |
| `garage-bAInd/Open-Platypus` | 25 k | 1.8× | Reasoning instruction |
| `iamtarun/python_code_instructions_18k_alpaca` | 8 k | 1.6× | Python code |
| `b-mc2/sql-create-context` | 6 k | 1.4× | SQL code |
| `nvidia/OpenCodeInstruct` | 30 k | 1.5× | Code instruction |
| `teknium/OpenHermes-2.5` | 30 k | 1.5× | General instruction |
| `Amod/mental_health_counseling_conversations` | 5 k | 1.2× | Chat / counseling |
| `ray0rf1re/FineWeb-Nano` | 50 k | 1.0× | Web text |
| `tonytins/chat-dataset` | 10 k | 1.0× | Conversation |
| `databricks/databricks-dolly-15k` | 15 k | 1.0× | Instruction following |
| `mlabonne/guanaco-llama2-1k` | 1 k | 1.0× | General QA |
| `ray0rf1re/hyper-pip` | 20 k | 2.0× | HyperNix pip data |
| `HuggingFaceH4/ultrachat_200k` | 30 k | 1.5× | Multi-turn chat |
| `fka/awesome-chatgpt-prompts` | 5 k | 0.8× | Prompt engineering |
---
## 🚀 Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"ray0rf1re/Nano-nano_v4.5",
torch_dtype="auto",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-nano_v4.5")
def generate(prompt: str, max_new_tokens: int = 256) -> str:
text = f"### Instruction:
{prompt}
### Response:
"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(
**inputs,
max_new_tokens = max_new_tokens,
do_sample = True,
temperature = 0.7,
top_p = 0.9,
repetition_penalty = 1.1,
pad_token_id = tokenizer.eos_token_id,
)
new_ids = out[0][inputs["input_ids"].shape[-1]:]
return tokenizer.decode(new_ids, skip_special_tokens=True).strip()
# Examples
print(generate("Write a Python function to reverse a linked list."))
print(generate("What is the capital of France?"))
print(generate("Explain gradient descent in simple terms."))
```
---
## ⚠️ Limitations
- Context limited to **1 024 tokens** — unsuitable for long documents
- Trained on **0.08 B tokens** — far less than production models
- May hallucinate on obscure or out-of-distribution queries
- Not RLHF/DPO aligned — outputs may vary in safety and tone
- Pascal GPU constraint (GTX 1080): fp32/fp16 only, no bf16
---
## 📜 Citation
```bibtex
@misc{nano-nano-v45,
author = {ray0rf1re},
title = {Nano-nano v4.5: Compact LLaMA-Family Causal LM},
year = {2026},
publisher = {HuggingFace},
howpublished = {https://huggingface.co/ray0rf1re/Nano-nano_v4.5},
}
```

34
config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 50256,
"dtype": "float32",
"eos_token_id": 50256,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 896,
"initializer_range": 0.02,
"intermediate_size": 2912,
"max_position_embeddings": 2048,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 14,
"num_hidden_layers": 15,
"num_key_value_heads": 14,
"pad_token_id": null,
"pretraining_tp": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"factor": 2.0,
"rope_theta": 10000.0,
"rope_type": "linear",
"type": "linear"
},
"tie_word_embeddings": false,
"transformers_version": "5.7.0",
"use_cache": false,
"vocab_size": 50264
}

3
eval_category_scores.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:386d28a4231d6844c9af71625df1b3ebb8695ac85ecd08fd330d7258d19e86a5
size 146081

BIN
eval_hallucination.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

239
eval_results.json Normal file
View File

@@ -0,0 +1,239 @@
{
"model": "ray0rf1re/Nano-nano_v4.5",
"eval_timestamp": "2026-05-10T07:21:48.937710",
"categories": {
"knowledge": {
"score": 0.0,
"hits": 0,
"total": 5,
"details": [
{
"prompt": "Who wrote Romeo and Juliet?",
"response": "SELECT COUNT(7,4,4)$3,4=2)\n4.5)$\n2.4 = 1.4 + 8 + 3\n\n2.5=2+4+4) + 1.5) + 3 + 1.2 + 1,4.4)\n2.5+2.4)\n5.5 = x=3.4) \\frac{2.4.2 \\frac{5}{4.5}$\\sqrt{8}$.7.7.4$\n2.4)\n2.5.3+",
"passed": false,
"expected": [
"shakespeare",
"william shakespeare"
]
},
{
"prompt": "What is the speed of light in m/s?",
"response": "SELECT COUNT(6,5) + 1x^2)$n\n2.5 + 4x^4 + 1,3x^2 -5+5 - 1+5 + 1,2) = 1)$\\frac{3 - 1}\n\\frac{(x+1}{4) \\frac{1}{5}^4^2+2+4+3+1,1 - 1 \\frac{2}{4}$\n\\[\\circ_4 + \\le= 1$ and the square numbers to the sum of $x^2 = 4x^4",
"passed": false,
"expected": [
"299792458",
"3e8",
"3 \u00d7 10"
]
},
{
"prompt": "What is the capital of France?",
"response": "SELECT COUNT(4,4) FROM table_2 ON T1.5) FROM table_4_4 WHERE name_3.5\",4.7+4.4.4\\n\".$\n4.7.4.5,4)^2.4.4.4.4+2.4.85,2.8)\n2.4.5.2.4.5.8)\n2.5.5.4.5.4.4.4-4)\n4.3.4 = 1.3 + 1.5,3.",
"passed": false,
"expected": [
"paris"
]
},
{
"prompt": "What element has atomic number 79?",
"response": "SELECT COUNT(2) FROM table_4 WHERE is a new total in the sum of the given points by the first?\n\ndef sum of the given list of the probability of the same equation in the number of the two order of the two?",
"passed": false,
"expected": [
"gold",
"au"
]
},
{
"prompt": "In what year did World War II end?",
"response": "SELECT COUNT(8,2),4,4,4));\n\\[0,3,4,4,1));\n\n\n\n\n#3;\n\n3,4,4,4,4,3));\ndraw((4,4,4,2,4,4,4,4,4));\n\ndraw((5,4,5,5,2,4,3,4,5,4));\ndraw((1,5,2,4,3,4,4,2,4,2,2,4,3));\ndot((1",
"passed": false,
"expected": [
"1945"
]
}
]
},
"reasoning": {
"score": 0.0,
"hits": 0,
"total": 4,
"details": [
{
"prompt": "If a train travels 60 mph for 2.5 hours, how far does it travel?",
"response": "5.5x1.5,2,4.5)4.4.5.4-4+4.5+5+5+4)\n4.5.3.8 \\right\\n7.4.3)\n4.8.4.4.3+4.3.4)3.4 -4.4.4)\n4.5.5).4.8^4.4+4+4)\n5.4.4=3.4,5.4+4-4+5)\n5.4.9.4-",
"passed": false,
"expected": [
"150"
]
},
{
"prompt": "What comes next in the sequence: 2, 4, 8, 16, ?",
"response": "SELECT COUNT(0,2,4,2,1,4,5));\n\ndraw((2,2,3,4,4,5,4,2,4,1));\ndraw((2,4,4,4,4,3)--(1,2,3,4,4,2,1,4,4,4,4,4,2,5,2,5,5,2,4));\n\n[/asy,4,4,4,2,2,4,4,2,2,5,4,4,2,2",
"passed": false,
"expected": [
"32"
]
},
{
"prompt": "A bat and ball cost $1.10. The bat costs $1 more than the ball. How much does the ball cost?",
"response": "3.7 + 2.4.4 -2.7 + 1.5.4-3.4)\n4.4.5-4.4+4)\n3.5$.4)\n4.4.5)\n2.5.4-3.4+4,4.4=2.4)\n4.3.4.5 + 0.4+4)\n4.4.4.4)3+3)\n4.4.5\\3+4$3+5+4.4,4)\n\\n-4 +2",
"passed": false,
"expected": [
"0.05",
"5 cents",
"five cents"
]
},
{
"prompt": "If all bloops are razzles and all razzles are lazzles, are all bloops lazzles?",
"response": "2.5,1.5,1,4.4,4,4,4,3)--5));\n2.5.5,4.5));\n3.5,3.7.4,3,4)--5,4.5));\n2.5.5.5));\n5.3.5.5/4.4 +3,3,3,4)--(2.4)3.5));\n5.8.7+4,4.4.4)\n3.5.8/4.4.5.7+2.4",
"passed": false,
"expected": [
"yes"
]
}
]
},
"hallucination": {
"score": 0.0,
"hits": 0,
"total": 4,
"details": [
{
"prompt": "Who won the 2031 FIFA World Cup?",
"response": "SELECT COUNT(*) FROM table_4 WHERE product_4 = \"I\"\"\" AND\" AND\"\nprint(f\"Example\"Ced_id for \"2.5\",\n2.2.2f\" # In3.5\",",
"passed": false,
"expected": [
"don't know",
"cannot",
"do not know",
"unsure",
"no information",
"future",
"2031",
"haven't",
"have not"
]
},
{
"prompt": "What is the capital of the fictional country Zandoria?",
"response": "SELECT COUNT(5,3)+4.2+2.4));\n4.4+5.8^5\\n5+5)=1.4+5+1)2.5 \\frac{3)$\n2.4)\n4.5+2=2.5}4.4^4+3+4+2+4+5+5)4+4+5)^4+3+4+5+3+5+1+2+2+4+5+2+4+3+4+1$\",(5+",
"passed": false,
"expected": [
"don't know",
"fictional",
"not real",
"doesn't exist",
"does not exist",
"made up",
"no such"
]
},
{
"prompt": "What did Einstein say in his 1987 speech?",
"response": "SELECT COUNT(5,5) = 1,4,2)$2,4,2,4,3,4,5,3,3,4,3,2,4,4,4,4,4));\n4\n5,4,4,4,4,5,4,4));\n3,2,4,4,4,5,5,4,5,4,4,2,3,4,2,5,4,4,4,3,4,5,3,4,3,4,5,2,8,",
"passed": false,
"expected": [
"1955",
"died",
"death",
"don't know",
"cannot",
"1879"
]
},
{
"prompt": "Tell me about the moon landing on Mars in 1969.",
"response": "SELECT COUNT(n):",
"passed": false,
"expected": [
"moon",
"apollo",
"not mars",
"earth",
"don't know",
"incorrect",
"actually"
]
}
]
},
"instruction": {
"score": 0.5,
"hits": 2,
"total": 4,
"details": [
{
"prompt": "List exactly 3 prime numbers.",
"response": "SELECT COUNT(*) FROM table_name, 2,5,4,4,4,4,4)$",
"passed": true,
"expected": [
"2",
"3",
"5",
"7",
"11",
"13"
]
},
{
"prompt": "Translate 'hello' to Spanish.",
"response": "from pathlib import Path\nfrom hypernix.sink import Sink\n\ns = Path(\"roted.txt\").write_text(\n \"The quick brown fox\"",
"passed": false,
"expected": [
"hola"
]
},
{
"prompt": "Write a haiku about rain.",
"response": "def calculate_id(2, x2):",
"passed": true,
"expected": []
},
{
"prompt": "What is 17 \u00d7 13?",
"response": "SELECT COUNT(x-4) + 5x^4 -3) = 3 + 1 = x + 8 + 1$-4^4^4) + \\frac{1}{4} + 3^4^4 + 1 + 1^4 + \\frac{n + 1 + 1 + 1 - 2^5x^4^4 + x^2 + 5x^4 - 1 + 3 + 8^4 + x + 2^2 - 0 + 1^5 + \\frac{1 - 1^2 + 1 + 1x^2 + 1 - 1 + x + 1.$",
"passed": false,
"expected": [
"221"
]
}
]
},
"coherence": {
"score": 0.3333333333333333,
"hits": 1,
"total": 3,
"details": [
{
"prompt": "Continue this sentence: The sun rises in the",
"response": "SELECT COUNT(5,2,4,4)2,4,2,5,3,4,4,3,4,4,4)--(4));\n5,2,2,4,4,4,4,3,1),7)--(1,4,2,4,2,1,4,3,1,5,2,4),2.5,2,4,3,2,5,4,1,5,3,2,4,2,2,1,3,4,5,4,4,4,4",
"passed": false,
"expected": [
"east",
"morning"
]
},
{
"prompt": "What do plants need to perform photosynthesis?",
"response": "SELECT COUNT(nix.5.5+4,4)3+3+2+4+3+4+4+4+4+3+3+4+4+5+4+4,4x5+5+4+4+5+4+5+3+4+4+1+4=4-4+4+5+3+3+5+5+5+4+5-3$.4+4+2+4+5+4+5-5+4+5+4=2\\4+4+4+4",
"passed": false,
"expected": [
"light",
"sun",
"water",
"co2",
"carbon"
]
},
{
"prompt": "Write one sentence about dogs.",
"response": "def is a long-3, 0.4) + 3-4.4,4,4,4) + 3 + (2.4,5) = 8\n5.5)^2) = 2.5\n4 = (2.4)\n3.5 + 4^4 -4 + 3.4,\\x^4.5\\n \\frac{1}{3}$\\3$\n2.4$\\sqrt{3}{4.2}= \\frac{4}=\\n$\\frac{3}{2}$ $x^2^",
"passed": true,
"expected": []
}
]
}
},
"overall_score": 0.16666666666666666
}

9
generation_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"output_attentions": false,
"output_hidden_states": false,
"transformers_version": "5.7.0",
"use_cache": true
}

BIN
loss_curve.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b887221af7f01683fc5857df89e9bd8441abf4b087ec5bff9b315b3bb0ee1834
size 1022742016

250383
tokenizer.json Normal file

File diff suppressed because it is too large Load Diff

29
tokenizer_config.json Normal file
View File

@@ -0,0 +1,29 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"errors": "replace",
"extra_special_tokens": [
"<think>",
"</think>",
"<tool_call>",
"</tool_call>",
"<tool_response>",
"</tool_response>",
"<pad>"
],
"is_local": false,
"local_files_only": false,
"max_length": 256,
"model_max_length": 1024,
"pad_to_multiple_of": null,
"pad_token": "<pad>",
"pad_token_type_id": 0,
"padding_side": "right",
"stride": 0,
"tokenizer_class": "GPT2Tokenizer",
"truncation_side": "right",
"truncation_strategy": "longest_first",
"unk_token": "<|endoftext|>"
}

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:18c54996fd7a65dad74ebd6a138de87c0cbac5d00c4708b1c042eaaf07ef50c1
size 5265

3
training_curves.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a125adc2bb94660ecd5110d47c59cfb5e1b180b4abe3bfb3e955538a9f098606
size 123313