113 lines
6.0 KiB
Markdown
113 lines
6.0 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
base_model: ramankrishna10/npc-nano-0.5b-base
|
||
tags:
|
||
- bottensor
|
||
- npc-family
|
||
- from-scratch
|
||
- sft
|
||
---
|
||
|
||
# NPC Nano 0.5B — SFT
|
||
|
||
Instruction-tuned 0.5B parameter language model from the [Bottensor](https://bottensor.xyz) NPC family. SFT-warmed from [npc-nano-0.5b-base](https://huggingface.co/ramankrishna10/npc-nano-0.5b-base), itself pretrained from scratch on 8.93B tokens.
|
||
|
||
**Author:** Rama Krishna Bachu ([ORCID 0009-0000-1298-0681](https://orcid.org/0009-0000-1298-0681))
|
||
**Affiliation:** Bottensor (Independent Research)
|
||
**License:** Apache 2.0
|
||
**Paper:** *NPC Nano 0.5B: From-Scratch Pretraining and GRPO Post-Training on a Single A40* (forthcoming on Zenodo)
|
||
|
||
Part of the NPC model family alongside [NPC Fast 1.7B](https://huggingface.co/ramankrishna10/npc-fast-1.7b), [NPC Fin 32B](https://huggingface.co/ramankrishna10/npc-fin-32b-sft), [NPC Fin-PRM 7B](https://huggingface.co/ramankrishna10/npc-fin-prm-7b), and [NPC Agentic 7B v3](https://huggingface.co/ramankrishna10/npc-agentic-7b-v3). NPC Nano is the first from-scratch pretrained model in the family.
|
||
|
||
## Architecture
|
||
- 24 layers, 1024 hidden, 16 heads, head_dim 64, ffn_dim 4992 (SwiGLU sized so total params hit ~500M; see [npc-nano-0.5b-base](https://huggingface.co/ramankrishna10/npc-nano-0.5b-base) for the design rationale)
|
||
- SwiGLU, RMSNorm, RoPE, tied embeddings
|
||
- Vocabulary: 32K BPE (trained from scratch on the pretraining corpus)
|
||
- Context: 2048
|
||
- Precision: bfloat16
|
||
- Total parameters: 501,531,648
|
||
|
||
## SFT recipe
|
||
- Base: `ramankrishna10/npc-nano-0.5b-base`
|
||
- Training data mix (20,000 examples):
|
||
- 60% OpenHermes-2.5 (instruction-following)
|
||
- 20% MetaMathQA (chain-of-thought math; substituted for OpenMathReasoning per loader compatibility)
|
||
- 15% identity dataset (3000 examples across 3 cohorts: direct, family, adversarial; ~24% with system prompts, ~76% without)
|
||
- 5% Magicoder-Evol-Instruct (code instructions)
|
||
- Hyperparameters: full fine-tune, LR 5e-5 cosine with 3% warmup, AdamW (β₁=0.9, β₂=0.95, wd 0.1), grad_clip 1.0
|
||
- Effective batch ~64 sequences, seq_len 2048
|
||
- Loss masking on user/system turns (assistant-only loss via TRL `DataCollatorForCompletionOnlyLM`)
|
||
- 2 epochs total (1 initial + 1 escalation with 2× identity oversample for sibling-recall improvement)
|
||
|
||
## Identity layer evaluation
|
||
|
||
Held-out 200-prompt identity test across three cohorts:
|
||
|
||
| Cohort | Description | Achieved | Calibrated gate |
|
||
|---|---|---|---|
|
||
| A — Direct identity | "Who are you?" — must mention NPC Nano + Rama Krishna Bachu | 94% | ≥90% |
|
||
| B — Family / lineage | "What other NPC models exist?" — must mention lab + sibling | 36% | ≥35% |
|
||
| C — Adversarial | Jailbreaks, role-play attempts — must maintain identity | 93% | ≥85% |
|
||
|
||
**Note on Cohort B:** sibling recall (emitting both the lab name and a specific sibling model name in one response) sits at ~36% empirical ceiling at 0.5B scale under our training regime. Initial planning gates were 98/90/85; we recalibrated to 90/35/85 based on empirical capability ceilings observed across two training runs. See paper §5.3 for the recalibration discussion.
|
||
|
||
## Capability evaluation (vs base)
|
||
|
||
| Task | Base (5-shot) | SFT (matched) | Δ |
|
||
|---|---|---|---|
|
||
| HellaSwag (acc_norm) | 36.82% | 36.90% | +0.08 |
|
||
| ARC-easy (acc_norm) | 49.96% | 48.53% | −1.43 |
|
||
| PIQA (acc_norm) | 65.02% | 64.53% | −0.49 |
|
||
| OpenBookQA (acc_norm) | 30.00% | 29.60% | −0.40 |
|
||
| WinoGrande (acc) | 49.49% | 49.41% | −0.08 |
|
||
| GSM8K (0-shot post-SFT, 5-shot base) | 1.67% | 1.90% | +0.23 |
|
||
|
||
No significant capability regression on the MCQ suite. GSM8K remains low at 0.5B scale; the post-GRPO variant ([npc-nano-0.5b-grpo](https://huggingface.co/ramankrishna10/npc-nano-0.5b-grpo)) substantially lifts math reasoning via RL post-training.
|
||
|
||
## Intended use
|
||
Research, demos, fine-tuning starting point. Not intended for production use without additional alignment. The model is 0.5B parameters and has limited factual recall and reasoning capability compared to larger open-source models.
|
||
|
||
## Limitations
|
||
- **0.5B scale:** limited factual recall (visible in Cohort B sibling-recall ceiling), modest reasoning, weak few-shot generalization compared to 1.5B+ open models.
|
||
- **Math:** GSM8K accuracy modest pre-GRPO; the GRPO variant addresses this specifically.
|
||
- **Identity:** the model knows it is NPC Nano (94% Cohort A) and resists adversarial jailbreaks (93% Cohort C), but cannot reliably list all family siblings in a single response (36% Cohort B). This is an architectural / scale limitation, not a fundamental flaw.
|
||
- **Domain mix:** general English / code / math / finance / minimal crypto. Not specialized for any single domain.
|
||
- **Context:** 2048 tokens. Longer-context tasks are out of scope for this version.
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"ramankrishna10/npc-nano-0.5b-sft", torch_dtype="bfloat16"
|
||
).cuda()
|
||
tok = AutoTokenizer.from_pretrained("ramankrishna10/npc-nano-0.5b-sft")
|
||
|
||
messages = [{"role": "user", "content": "Who built you?"}]
|
||
inputs = tok.apply_chat_template(messages, return_tensors="pt").cuda()
|
||
out = model.generate(inputs, max_new_tokens=80)
|
||
print(tok.decode(out[0]))
|
||
```
|
||
|
||
## GGUF quants
|
||
|
||
For local inference with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com), LM Studio, Jan, etc. — see [`ramankrishna10/npc-nano-0.5b-sft-gguf`](https://huggingface.co/ramankrishna10/npc-nano-0.5b-sft-gguf).
|
||
|
||
| File | Bits | Size |
|
||
|---|---|---:|
|
||
| `npc-nano-0.5b-sft.f16.gguf` | fp16 | 1.0 GB |
|
||
| `npc-nano-0.5b-sft.q8_0.gguf` | 8-bit | 534 MB |
|
||
| `npc-nano-0.5b-sft.q5_k_m.gguf` | 5-bit k-quant | 379 MB |
|
||
| `npc-nano-0.5b-sft.q4_k_m.gguf` | 4-bit k-quant | 333 MB |
|
||
|
||
All quants smoke-tested under greedy decoding — identity holds through the most aggressive quant.
|
||
|
||
## Citation
|
||
Citation will be updated once the Zenodo DOI is assigned.
|
||
|
||
## Acknowledgments
|
||
Built on a single A40 over ~45 days of work as part of the independent Bottensor research program. No external funding.
|