初始化项目，由ModelHub XC社区提供模型

Model: KookiesXy/Neo50M Source: Original Platform
2026-07-17 09:27:11 +08:00
commit 5f250fbae2
22 changed files with 277683 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,39 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+gguf/neo50m-f16.gguf filter=lfs diff=lfs merge=lfs -text
+gguf/neo50m-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+gguf/neo50m-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+onnx/model.onnx.data filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,72 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- llama
+- causal-lm
+- chat
+- tiny-language-model
+- neo50m
+---
+
+# Neo50M
+
+Neo50M is a tiny decoder-only chat language model trained from scratch. It is designed for toy/local assistant use, educational experiments, lightweight generation, and testing training pipelines.
+
+## Model Details
+
+- **Type:** decoder-only causal language model, Llama-compatible architecture
+- **Parameters:** approximately 52.6M
+- **Context length target:** 16k tokens
+- **Training target:** about 15B pretraining tokens plus chat/instruction tuning
+- **Hardware:** 8x NVIDIA RTX 5090 cloud GPUs
+- **Tokenizer:** TinyLlama/Llama-style 32k tokenizer with a Neo50M chat template
+
+## Intended Uses
+
+- toy/local assistant experiments
+- educational training and inference demos
+- lightweight generation
+- testing HF, GGUF, ONNX, and distributed training pipelines
+
+## Limitations
+
+Neo50M is very small. It is not reliable for factual accuracy, has limited reasoning ability, may hallucinate, and should not be used for safety-critical decisions or high-stakes advice.
+
+## Transformers Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "KookiesXy/Neo50M"
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForCausalLM.from_pretrained(repo_id, device_map="auto")
+
+messages = [{"role": "user", "content": "Write a short thank-you note."}]
+inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
+out = model.generate(inputs, max_new_tokens=120, temperature=0.7, top_p=0.9)
+print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+## GGUF Usage
+
+After downloading a GGUF file:
+
+```bash
+llama-cli -m neo50m-q4_k_m.gguf -p "User: Write a haiku about GPUs.\nAssistant:"
+```
+
+## ONNX Usage
+
+The ONNX export is intended for forward-pass validation and integration experiments. Use ONNX Runtime to load `onnx/model.onnx` and feed integer `input_ids` plus `attention_mask`.
+
+## Dataset Summary
+
+The training pipeline streams a configurable mixture of FineWeb-Edu, Cosmopedia, Wikipedia-like text, TinyStories, and a small permissive code component. SFT uses OpenHermes-style, UltraChat-style, Alpaca-style, and small refusal/helpfulness examples when available. Dataset availability can change; the exact configs are included with the upload.
+
+## Eval Results
+
+Eval artifacts, when present, are uploaded under `evals/`.
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1 @@
+{% for message in messages %}{% if loop.first and message['role'] != 'system' %}{{ bos_token + 'System: You are Neo50M, a concise and helpful assistant.\n' }}{% endif %}{% if message['role'] == 'system' %}{{ bos_token + 'System: ' + message['content'].strip() + '\n' }}{% elif message['role'] == 'user' %}{{ 'User: ' + message['content'].strip() + '\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'].strip() + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant: ' }}{% endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "max_position_embeddings": 16384,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 2,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "use_cache": true,
+  "vocab_size": 32000
+}
--- a/evals/eval_results.json
+++ b/evals/eval_results.json
@@ -0,0 +1,33 @@
+{
+  "validation": {
+    "validation_loss": 3.467215418815613,
+    "validation_perplexity": 32.047379553269614
+  },
+  "prompts": [
+    {
+      "prompt": "What is Neo50M?",
+      "response": "1."
+    },
+    {
+      "prompt": "Write a concise email asking to reschedule a meeting.",
+      "response": "1.2.1.2.2.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3."
+    },
+    {
+      "prompt": "Explain why the sky appears blue in two sentences.",
+      "response": "1. The first sentence is called 'a' and the second sentence is called 'a'."
+    },
+    {
+      "prompt": "Write a Python function that returns the factorial of a non-negative integer.",
+      "response": "2.2.3\n\"\"\"\n\n    return int(x)"
+    },
+    {
+      "prompt": "If a train travels 60 miles in 90 minutes, what is its average speed in miles per hour?",
+      "response": "200 minutes, it is a track for the energy of the train, and it is also a track for the 60 minutes. The train will then be used to increase the speed of"
+    }
+  ],
+  "long_context_smoke": {
+    "ok": true,
+    "tokens": 4096,
+    "last_logit_norm": 596.793701171875
+  }
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,10 @@
+{
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": 2,
+  "max_new_tokens": 256,
+  "pad_token_id": 2,
+  "temperature": 0.7,
+  "top_p": 0.9,
+  "transformers_version": "5.12.1"
+}
--- a/gguf/neo50m-f16.gguf
+++ b/gguf/neo50m-f16.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:019ac05a6faa53a813d5d775f1e0b3e364ca88361f09a440da01bfddea009182
+size 105902496
--- a/gguf/neo50m-q4_k_m.gguf
+++ b/gguf/neo50m-q4_k_m.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a997dbfb928db1a6fc6a553afea1d204000e8dd4dd13c48ec2c1c55bd4f2a96
+size 35889568
--- a/gguf/neo50m-q8_0.gguf
+++ b/gguf/neo50m-q8_0.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:884ca9915dd0b67ac880adf3ba0c849f2a9b85b231883768278e8f2e1fc7df4d
+size 56627616
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc82614943d98a47f0a93bf0ed3748325ce394162fb06a12a235983c79df1db9
+size 210302864
--- a/onnx/model.onnx
+++ b/onnx/model.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4984dc32baf632968f94f936c1d39d5e60197b5ba76932e40dca95a2f00b5bdb
+size 1537878
--- a/onnx/model.onnx.data
+++ b/onnx/model.onnx.data
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cce8d00c4f02e58e69dcabcbe89336306cfb13027e6042073cf3c9e108dd8c0c
+size 210305024
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,16 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 16384,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}
--- a/training_configs/data_mix.yaml
+++ b/training_configs/data_mix.yaml
@@ -0,0 +1,85 @@
+pretrain:
+  shuffle_buffer: 10000
+  seed: 1337
+  allow_synthetic_fallback: false
+  max_doc_tokens: 16384
+  max_chars_per_doc: 120000
+  sources:
+    - name: fineweb_edu
+      path: HuggingFaceFW/fineweb-edu
+      config: sample-10BT
+      split: train
+      text_field: text
+      weight: 0.0
+      streaming: true
+      fail_on_error: false
+    - name: cosmopedia
+      path: HuggingFaceTB/cosmopedia
+      config: web_samples_v2
+      split: train
+      text_field: text
+      weight: 0.0
+      streaming: true
+      fail_on_error: false
+    - name: wikipedia
+      path: wikimedia/wikipedia
+      config: 20231101.en
+      split: train
+      text_field: text
+      weight: 0.0
+      streaming: true
+      fail_on_error: false
+    - name: tinystories
+      path: roneneldan/TinyStories
+      split: train
+      text_field: text
+      weight: 0.75
+      streaming: true
+      fail_on_error: false
+    - name: permissive_code
+      path: nuprl/stack-dedup-python-testgen-starcoder-filter-v2
+      split: train
+      text_field: content
+      weight: 0.25
+      streaming: true
+      fail_on_error: false
+
+validation:
+  sources:
+    - name: wikitext103
+      path: Salesforce/wikitext
+      config: wikitext-103-raw-v1
+      split: validation
+      text_field: text
+      weight: 1.0
+      streaming: true
+      fail_on_error: false
+
+sft:
+  shuffle_buffer: 5000
+  seed: 2025
+  assistant_only_loss: true
+  allow_synthetic_fallback: true
+  sources:
+    - name: openhermes
+      path: teknium/OpenHermes-2.5
+      split: train
+      weight: 0.45
+      streaming: false
+      fail_on_error: false
+    - name: ultrachat
+      path: HuggingFaceH4/ultrachat_200k
+      split: train_sft
+      weight: 0.35
+      streaming: false
+      fail_on_error: false
+    - name: alpaca
+      path: tatsu-lab/alpaca
+      split: train
+      weight: 0.15
+      streaming: false
+      fail_on_error: false
+    - name: safety_small
+      synthetic: safety_refusals
+      weight: 0.05
+      fail_on_error: false
--- a/training_configs/export.yaml
+++ b/training_configs/export.yaml
@@ -0,0 +1,12 @@
+export:
+  repo_id: KookiesXy/Neo50M
+  checkpoint: outputs/checkpoints/best
+  hf_dir: outputs/hf
+  onnx_dir: outputs/onnx
+  gguf_dir: outputs/gguf
+  eval_dir: outputs/evals
+  max_shard_size: 2GB
+  gguf_outtypes:
+    - f16
+    - q8_0
+    - q4_k_m
--- a/training_configs/long_context.yaml
+++ b/training_configs/long_context.yaml
@@ -0,0 +1,31 @@
+run:
+  name: neo50m_long_context
+  output_dir: outputs
+  model_config: configs/model_neo50m.yaml
+  data_config: configs/data_mix.yaml
+  seed: 2337
+
+training:
+  stage: long_context
+  target_tokens: 600000000
+  micro_batch_size: 1
+  grad_accum: 16
+  learning_rate: 0.0002
+  min_lr_ratio: 0.2
+  warmup_steps: 500
+  weight_decay: 0.1
+  beta1: 0.9
+  beta2: 0.95
+  grad_clip: 1.0
+  eval_interval_steps: 500
+  eval_batches: 8
+  save_interval_steps: 500
+  keep_last_checkpoints: 2
+  log_interval_steps: 5
+  num_workers: 0
+  data_mode: streaming
+  curriculum:
+    - seq_len: 8192
+      tokens: 300000000
+    - seq_len: 16384
+      tokens: 300000000
--- a/training_configs/model_neo50m.yaml
+++ b/training_configs/model_neo50m.yaml
@@ -0,0 +1,21 @@
+model:
+  name: Neo50M
+  architecture: llama
+  tokenizer_id: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+  hidden_size: 512
+  intermediate_size: 1536
+  num_hidden_layers: 12
+  num_attention_heads: 8
+  num_key_value_heads: 2
+  max_position_embeddings: 16384
+  rope_theta: 500000.0
+  rms_norm_eps: 0.000001
+  initializer_range: 0.02
+  tie_word_embeddings: true
+  use_cache: false
+  attention_dropout: 0.0
+  attention_bias: false
+  mlp_bias: false
+  attn_implementation: sdpa
+  gradient_checkpointing: false
+  torch_dtype: bfloat16
--- a/training_configs/pretrain_15b.yaml
+++ b/training_configs/pretrain_15b.yaml
@@ -0,0 +1,27 @@
+run:
+  name: neo50m_pretrain_15b
+  output_dir: outputs
+  model_config: configs/model_neo50m.yaml
+  data_config: configs/data_mix.yaml
+  seed: 1337
+
+training:
+  stage: pretrain_4k
+  seq_len: 2048
+  target_tokens: 15000000000
+  micro_batch_size: 4
+  grad_accum: 4
+  learning_rate: 0.0008
+  min_lr_ratio: 0.1
+  warmup_steps: 2000
+  weight_decay: 0.1
+  beta1: 0.9
+  beta2: 0.95
+  grad_clip: 1.0
+  eval_interval_steps: 1000
+  eval_batches: 16
+  save_interval_steps: 1000
+  keep_last_checkpoints: 2
+  log_interval_steps: 10
+  num_workers: 0
+  data_mode: streaming
--- a/training_configs/pretrain_fast.yaml
+++ b/training_configs/pretrain_fast.yaml
@@ -0,0 +1,27 @@
+run:
+  name: neo50m_smoke
+  output_dir: outputs/smoke
+  model_config: configs/model_neo50m.yaml
+  data_config: configs/data_mix.yaml
+  seed: 1337
+
+training:
+  stage: smoke
+  seq_len: 256
+  target_tokens: 1000000
+  micro_batch_size: 2
+  grad_accum: 1
+  learning_rate: 0.0006
+  min_lr_ratio: 0.1
+  warmup_steps: 20
+  weight_decay: 0.1
+  beta1: 0.9
+  beta2: 0.95
+  grad_clip: 1.0
+  eval_interval_steps: 0
+  eval_batches: 2
+  save_interval_steps: 100
+  keep_last_checkpoints: 2
+  log_interval_steps: 10
+  num_workers: 0
+  data_mode: random
--- a/training_configs/sft_chat.yaml
+++ b/training_configs/sft_chat.yaml
@@ -0,0 +1,34 @@
+run:
+  name: neo50m_sft_chat
+  output_dir: outputs
+  model_config: configs/model_neo50m.yaml
+  data_config: configs/data_mix.yaml
+  seed: 3337
+
+training:
+  stage: sft_chat
+  seq_len: 4096
+  target_tokens: 80000000
+  micro_batch_size: 1
+  grad_accum: 8
+  learning_rate: 0.00008
+  min_lr_ratio: 0.1
+  warmup_steps: 200
+  weight_decay: 0.0
+  beta1: 0.9
+  beta2: 0.95
+  grad_clip: 1.0
+  eval_interval_steps: 250
+  eval_batches: 4
+  save_interval_steps: 250
+  keep_last_checkpoints: 2
+  log_interval_steps: 5
+  num_workers: 0
+  data_mode: sft
+  curriculum:
+    - seq_len: 4096
+      tokens: 40000000
+    - seq_len: 8192
+      tokens: 25000000
+    - seq_len: 16384
+      tokens: 15000000
--- a/training_metadata.json
+++ b/training_metadata.json
@@ -0,0 +1,96 @@
+{
+  "best_val_loss": 2.788245379924774,
+  "config": {
+    "_config_path": "configs/sft_chat.yaml",
+    "run": {
+      "data_config": "configs/data_mix.yaml",
+      "model_config": "configs/model_neo50m.yaml",
+      "name": "neo50m_sft_chat",
+      "output_dir": "outputs",
+      "seed": 3337
+    },
+    "training": {
+      "beta1": 0.9,
+      "beta2": 0.95,
+      "curriculum": [
+        {
+          "seq_len": 4096,
+          "tokens": 40000000
+        },
+        {
+          "seq_len": 8192,
+          "tokens": 25000000
+        },
+        {
+          "seq_len": 16384,
+          "tokens": 15000000
+        }
+      ],
+      "data_mode": "sft",
+      "eval_batches": 4,
+      "eval_interval_steps": 250,
+      "grad_accum": 8,
+      "grad_clip": 1.0,
+      "keep_last_checkpoints": 2,
+      "learning_rate": 8e-05,
+      "log_interval_steps": 5,
+      "micro_batch_size": 1,
+      "min_lr_ratio": 0.1,
+      "num_workers": 0,
+      "save_interval_steps": 250,
+      "seq_len": 4096,
+      "stage": "sft_chat",
+      "target_tokens": 80000000,
+      "warmup_steps": 200,
+      "weight_decay": 0.0
+    }
+  },
+  "env": {
+    "cuda": "13.0",
+    "git_revision": null,
+    "gpu_count": 8,
+    "gpu_name": "NVIDIA GeForce RTX 5090",
+    "python": "3.12.13",
+    "torch": "2.12.0+cu130"
+  },
+  "global_step": 57750,
+  "global_tokens": 15628763136,
+  "model_config": {
+    "architecture": "llama",
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_implementation": "sdpa",
+    "gradient_checkpointing": false,
+    "hidden_size": 512,
+    "initializer_range": 0.02,
+    "intermediate_size": 1536,
+    "max_position_embeddings": 16384,
+    "mlp_bias": false,
+    "name": "Neo50M",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 12,
+    "num_key_value_heads": 2,
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 500000.0,
+    "tie_word_embeddings": true,
+    "tokenizer_id": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+    "torch_dtype": "bfloat16",
+    "use_cache": false
+  },
+  "precision": {
+    "fp8_backend": null,
+    "reason": "FP8 unavailable; torchao is not installed",
+    "requested": "auto",
+    "selected": "bf16"
+  },
+  "stage_id": "sft_chat_0_seq4096",
+  "stage_tokens": 25690112,
+  "tokenizer": {
+    "bos_token_id": 1,
+    "chat_template": "{% for message in messages %}{% if loop.first and message['role'] != 'system' %}{{ bos_token + 'System: You are Neo50M, a concise and helpful assistant.\\n' }}{% endif %}{% if message['role'] == 'system' %}{{ bos_token + 'System: ' + message['content'].strip() + '\\n' }}{% elif message['role'] == 'user' %}{{ 'User: ' + message['content'].strip() + '\\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'].strip() + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant: ' }}{% endif %}",
+    "eos_token_id": 2,
+    "pad_token_id": 2,
+    "tokenizer_id": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+    "vocab_size": 32000
+  }
+}
				`@@ -0,0 +1 @@`
				{% for message in messages %}{% if loop.first and message['role'] != 'system' %}{{ bos_token + 'System: You are Neo50M, a concise and helpful assistant.\n' }}{% endif %}{% if message['role'] == 'system' %}{{ bos_token + 'System: ' + message['content'].strip() + '\n' }}{% elif message['role'] == 'user' %}{{ 'User: ' + message['content'].strip() + '\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'].strip() + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant: ' }}{% endif %}