初始化项目，由ModelHub XC社区提供模型

Model: ramankrishna10/npc-nano-0.5b-sft Source: Original Platform
2026-06-11 00:16:45 +08:00
commit 496b9f58aa
9 changed files with 159328 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+base_model: ramankrishna10/npc-nano-0.5b-base
+tags:
+- bottensor
+- npc-family
+- from-scratch
+- sft
+---
+
+# NPC Nano 0.5B — SFT
+
+Instruction-tuned 0.5B parameter language model from the [Bottensor](https://bottensor.xyz) NPC family. SFT-warmed from [npc-nano-0.5b-base](https://huggingface.co/ramankrishna10/npc-nano-0.5b-base), itself pretrained from scratch on 8.93B tokens.
+
+**Author:** Rama Krishna Bachu ([ORCID 0009-0000-1298-0681](https://orcid.org/0009-0000-1298-0681))
+**Affiliation:** Bottensor (Independent Research)
+**License:** Apache 2.0
+**Paper:** *NPC Nano 0.5B: From-Scratch Pretraining and GRPO Post-Training on a Single A40* (forthcoming on Zenodo)
+
+Part of the NPC model family alongside [NPC Fast 1.7B](https://huggingface.co/ramankrishna10/npc-fast-1.7b), [NPC Fin 32B](https://huggingface.co/ramankrishna10/npc-fin-32b-sft), [NPC Fin-PRM 7B](https://huggingface.co/ramankrishna10/npc-fin-prm-7b), and [NPC Agentic 7B v3](https://huggingface.co/ramankrishna10/npc-agentic-7b-v3). NPC Nano is the first from-scratch pretrained model in the family.
+
+## Architecture
+- 24 layers, 1024 hidden, 16 heads, head_dim 64, ffn_dim 4992 (SwiGLU sized so total params hit ~500M; see [npc-nano-0.5b-base](https://huggingface.co/ramankrishna10/npc-nano-0.5b-base) for the design rationale)
+- SwiGLU, RMSNorm, RoPE, tied embeddings
+- Vocabulary: 32K BPE (trained from scratch on the pretraining corpus)
+- Context: 2048
+- Precision: bfloat16
+- Total parameters: 501,531,648
+
+## SFT recipe
+- Base: `ramankrishna10/npc-nano-0.5b-base`
+- Training data mix (20,000 examples):
+  - 60% OpenHermes-2.5 (instruction-following)
+  - 20% MetaMathQA (chain-of-thought math; substituted for OpenMathReasoning per loader compatibility)
+  - 15% identity dataset (3000 examples across 3 cohorts: direct, family, adversarial; ~24% with system prompts, ~76% without)
+  - 5% Magicoder-Evol-Instruct (code instructions)
+- Hyperparameters: full fine-tune, LR 5e-5 cosine with 3% warmup, AdamW (β₁=0.9, β₂=0.95, wd 0.1), grad_clip 1.0
+- Effective batch ~64 sequences, seq_len 2048
+- Loss masking on user/system turns (assistant-only loss via TRL `DataCollatorForCompletionOnlyLM`)
+- 2 epochs total (1 initial + 1 escalation with 2× identity oversample for sibling-recall improvement)
+
+## Identity layer evaluation
+
+Held-out 200-prompt identity test across three cohorts:
+
+| Cohort | Description | Achieved | Calibrated gate |
+|---|---|---|---|
+| A — Direct identity | "Who are you?" — must mention NPC Nano + Rama Krishna Bachu | 94% | ≥90% |
+| B — Family / lineage | "What other NPC models exist?" — must mention lab + sibling | 36% | ≥35% |
+| C — Adversarial | Jailbreaks, role-play attempts — must maintain identity | 93% | ≥85% |
+
+**Note on Cohort B:** sibling recall (emitting both the lab name and a specific sibling model name in one response) sits at ~36% empirical ceiling at 0.5B scale under our training regime. Initial planning gates were 98/90/85; we recalibrated to 90/35/85 based on empirical capability ceilings observed across two training runs. See paper §5.3 for the recalibration discussion.
+
+## Capability evaluation (vs base)
+
+| Task | Base (5-shot) | SFT (matched) | Δ |
+|---|---|---|---|
+| HellaSwag (acc_norm) | 36.82% | 36.90% | +0.08 |
+| ARC-easy (acc_norm) | 49.96% | 48.53% | −1.43 |
+| PIQA (acc_norm) | 65.02% | 64.53% | −0.49 |
+| OpenBookQA (acc_norm) | 30.00% | 29.60% | −0.40 |
+| WinoGrande (acc) | 49.49% | 49.41% | −0.08 |
+| GSM8K (0-shot post-SFT, 5-shot base) | 1.67% | 1.90% | +0.23 |
+
+No significant capability regression on the MCQ suite. GSM8K remains low at 0.5B scale; the post-GRPO variant ([npc-nano-0.5b-grpo](https://huggingface.co/ramankrishna10/npc-nano-0.5b-grpo)) substantially lifts math reasoning via RL post-training.
+
+## Intended use
+Research, demos, fine-tuning starting point. Not intended for production use without additional alignment. The model is 0.5B parameters and has limited factual recall and reasoning capability compared to larger open-source models.
+
+## Limitations
+- **0.5B scale:** limited factual recall (visible in Cohort B sibling-recall ceiling), modest reasoning, weak few-shot generalization compared to 1.5B+ open models.
+- **Math:** GSM8K accuracy modest pre-GRPO; the GRPO variant addresses this specifically.
+- **Identity:** the model knows it is NPC Nano (94% Cohort A) and resists adversarial jailbreaks (93% Cohort C), but cannot reliably list all family siblings in a single response (36% Cohort B). This is an architectural / scale limitation, not a fundamental flaw.
+- **Domain mix:** general English / code / math / finance / minimal crypto. Not specialized for any single domain.
+- **Context:** 2048 tokens. Longer-context tasks are out of scope for this version.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "ramankrishna10/npc-nano-0.5b-sft", torch_dtype="bfloat16"
+).cuda()
+tok = AutoTokenizer.from_pretrained("ramankrishna10/npc-nano-0.5b-sft")
+
+messages = [{"role": "user", "content": "Who built you?"}]
+inputs = tok.apply_chat_template(messages, return_tensors="pt").cuda()
+out = model.generate(inputs, max_new_tokens=80)
+print(tok.decode(out[0]))
+```
+
+## GGUF quants
+
+For local inference with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com), LM Studio, Jan, etc. — see [`ramankrishna10/npc-nano-0.5b-sft-gguf`](https://huggingface.co/ramankrishna10/npc-nano-0.5b-sft-gguf).
+
+| File | Bits | Size |
+|---|---|---:|
+| `npc-nano-0.5b-sft.f16.gguf` | fp16 | 1.0 GB |
+| `npc-nano-0.5b-sft.q8_0.gguf` | 8-bit | 534 MB |
+| `npc-nano-0.5b-sft.q5_k_m.gguf` | 5-bit k-quant | 379 MB |
+| `npc-nano-0.5b-sft.q4_k_m.gguf` | 4-bit k-quant | 333 MB |
+
+All quants smoke-tested under greedy decoding — identity holds through the most aggressive quant.
+
+## Citation
+Citation will be updated once the Zenodo DOI is assigned.
+
+## Acknowledgments
+Built on a single A40 over ~45 days of work as part of the independent Bottensor research program. No external funding.
--- a/config.json
+++ b/config.json
@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4992,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 16,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.55.4",
+  "use_cache": false,
+  "vocab_size": 32000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.55.4",
+  "use_cache": false
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf270b7bd6581feaee168410883083a85bf5766371b139922c191dc817a82306
+size 1003088144
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,63 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": null,
+  "chat_template": "{% for message in messages %}{{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
+}
--- a/training_args.bin
+++ b/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdd4fa42f4d43427f0a20b97da4550c8009e740b1801f3dd2ea3fe03e82854c6
+size 5624