初始化项目，由ModelHub XC社区提供模型

Model: Yusiko/khazri-2-mini Source: Original Platform
2026-09-11 15:14:18 +08:00
commit a74031e5cf
13 changed files with 163392 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/khazri-2-mini-banner.png filter=lfs diff=lfs merge=lfs -text
+assets/khazri-icon.png filter=lfs diff=lfs merge=lfs -text
+assets/khazri-wordmark.png filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,143 @@
+---
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb
+- bigcode/the-stack-dedup
+- GAIR/lima
+language:
+- en
+pipeline_tag: text-classification
+tags:
+- khazri
+- softyu
+- ai
+- llm
+- azerbaijani-national-llm
+- azerbaijan
+---
+<p align="center">
+  <img src="./assets/khazri-2-mini-banner.png" alt="Khazri 2 Mini — compact open-weight language model" width="100%">
+</p>
+
+# Khazri 2 Mini — 100M
+
+**Khazri 2 Mini** is a compact, open-weight decoder-only language model in the Khazri family. It combines a modern LLaMA-style Transformer with a custom Byte-Level BPE tokenizer and a 2B-token English training corpus.
+
+[Hugging Face](https://huggingface.co/Yusiko/khazri-2-mini) · [Khazri](https://khazri.dev) · [Contact](mailto:contact@khazri.dev)
+
+## At a glance
+
+| Item | Detail |
+| --- | --- |
+| Model | Khazri 2 Mini |
+| Parameters | **100.68M** |
+| Status | Open weights on Hugging Face |
+| Architecture | LLaMA-style, decoder-only Transformer |
+| Training precision | bf16 |
+| Context configured for training | 1,024 tokens |
+| Vocabulary | 32,768 tokens |
+| Tokenizer | Custom Byte-Level BPE |
+| Attention | Grouped-Query Attention: 12 query heads / 4 KV heads |
+
+## Architecture
+
+| Component | Configuration |
+| --- | --- |
+| Hidden size | 768 |
+| Transformer layers | 12 |
+| Attention heads | 12 |
+| Key/value heads | 4 |
+| MLP intermediate size | 2,048 |
+| Positional encoding | RoPE |
+| Normalization | RMSNorm |
+| MLP activation | SwiGLU / SiLU |
+| Attention backend | FlashAttention-2 where available; PyTorch SDPA fallback |
+| Embeddings | Tied input/output embeddings |
+
+## Training data
+
+Khazri 2 Mini is trained on a custom, pretokenized **English-only** corpus with a target size of **2,000,000,000 tokens**. The corpus is packed into **1,953,125 sequences** of 1,024 tokens and stored in Arrow shards with source identifiers.
+
+The documented token budget is:
+
+| Source | Token budget | Share | Role |
+| --- | ---: | ---: | --- |
+| [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) | 850M | 42.5% | General English educational and synthetic-text coverage |
+| [The Stack v2 Dedup](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) | 450M | 22.5% | Code from Python, JavaScript, TypeScript, Java, C++, C, Go and Rust |
+| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) | 300M | 15.0% | Simple narrative language |
+| [LIMA](https://huggingface.co/datasets/GAIR/lima) | 50M | 2.5% | Instruction and conversation examples |
+| SYNAPSE synthetic instruction data | 350M | 17.5% | Arithmetic, context, abstention, web-needed, identity, symbolic-math and general-assistant routes |
+
+The listed values are the documented source-token budget. The release manifest should be used for the final source counts of a particular weight revision.
+
+### Data processing and safeguards
+
+- Only English text is retained for this corpus. Short records are excluded, very long records are capped, and language/character checks are applied before tokenization.
+- The code portion is limited to the eight languages listed above.
+- The training mix combines general English, narrative, code, instruction and route-aware synthetic material. This preserves general capabilities while teaching specialized SYNAPSE behaviours.
+- Checkpoint evaluation should cover general Q&A, code, arithmetic, context extraction, abstention and current-information requests. A route-specific gain should not be accepted if it degrades general behaviour.
+- Original source datasets remain subject to their own terms and licences. Consult their source pages and the model repository licence before use.
+
+## Tokenizer
+
+Khazri 2 Mini uses a 32,768-token custom Byte-Level BPE tokenizer. It reserves structural whitespace, chat and SYNAPSE route tokens as single tokens, preserves indentation for code, and uses single-digit splitting to make arithmetic strings more explicit to the model.
+
+## Installation
+
+~~~bash
+pip install -U torch transformers accelerate safetensors
+~~~
+
+## Quick start
+
+~~~python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "Yusiko/khazri-2-mini"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype="auto",
+    device_map="auto",
+)
+
+prompt = "Write a concise explanation of a small language model."
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=160,
+        do_sample=False,
+        pad_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+~~~
+
+## Khazri 2 Preview comparison
+
+The following table reports the project-provided compact-model comparison for **Khazri 2 Preview**, not Khazri 2 Mini. Higher is better for every listed task.
+
+| Model | Parameters | Context extraction | Mixed speed/proxy | Arithmetic | Word problems | Abstention |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| **Khazri 2 Preview** | ~250M | **100%** | **62%** | **99%** | **99%** | **97.4%** |
+| Gemma 3 | 270M | 100% | 36% | 0% | 0% | 18% |
+| Qwen 2.5 | 0.5B | 89% | 45% | 14% | 28% | 46% |
+| Pythia | 160M | 22% | 10% | 0% | 2% | 1% |
+
+These are internal preview results on selected compact-model tests. They are not independently audited and should not be used to make claims about Khazri 2 Mini. Publish prompts, model revisions, scoring rules, hardware and complete evaluation assets with any future benchmark announcement.
+
+## Responsible use
+
+Khazri 2 Mini can produce incorrect, incomplete or biased outputs. Evaluate it on your own task, verify material claims and keep a human in the loop for consequential decisions. Do not rely on it as the sole basis for legal, medical, financial, hiring, safety or other high-impact decisions.
+
+## Roadmap
+
+Khazri 2 Mini is part of the second Khazri generation. The next planned stage is **Khazri 3**: a larger parameter scale and stronger results.
+
+## Contact
+
+For research, integration or partnership inquiries, visit [khazri.dev](https://khazri.dev) or email [contact@khazri.dev](mailto:contact@khazri.dev).
--- a/assets/image.png
+++ b/assets/image.png
--- a/assets/khazri-2-mini-banner.png
+++ b/assets/khazri-2-mini-banner.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e4d70b1b34d889ddaafe610401cd18827c2c8ede295305f20a20134f2bf50a5
+size 1687929
--- a/assets/khazri-icon.png
+++ b/assets/khazri-icon.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77b10a8a34b8d2921d68d4f899dd588237b23a1016c395bf9892f3260cd5971e
+size 875047
--- a/assets/khazri-wordmark.png
+++ b/assets/khazri-wordmark.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:044859ccb56b4d4b8d0d17d5a12014256274051ac69b75ffc3f59f6e6abc407e
+size 870591
--- a/config.json
+++ b/config.json
@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 4,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.10.2",
+  "use_cache": false,
+  "vocab_size": 32768
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 2,
+  "transformers_version": "5.10.2",
+  "use_cache": false
+}
--- a/khazri_tokenizer_runtime.py
+++ b/khazri_tokenizer_runtime.py
@@ -0,0 +1,41 @@
+
+import re
+
+def normalize_text(text: str) -> str:
+    if not isinstance(text, str):
+        return ""
+    text = text.replace("\r\n", "\n").replace("\r", "\n")
+    text = re.sub(r"[ \t]+$", "", text, flags=re.MULTILINE)
+    text = re.sub(r"\n{5,}", "\n\n\n\n", text)
+    return text.strip()
+
+def encode_structural_whitespace(text: str) -> str:
+    text = normalize_text(text)
+    if not text:
+        return ""
+    out_lines = []
+    for line in text.split("\n"):
+        line = line.replace("\t", " <|tab|> ")
+        m = re.match(r"^( +)", line)
+        if m:
+            n = len(m.group(1))
+            rest = line[n:]
+            tags = []
+            while n >= 4:
+                tags.append("<|indent_4|>")
+                n -= 4
+            while n >= 2:
+                tags.append("<|indent_2|>")
+                n -= 2
+            if n == 1:
+                rest = " " + rest
+            line = (" ".join(tags) + (" " if tags and rest else "") + rest)
+        out_lines.append(line)
+    return " <|nl|> ".join(out_lines).strip()
+
+def decode_structural_whitespace(text: str) -> str:
+    text = text.replace(" <|nl|> ", "\n").replace("<|nl|>", "\n")
+    text = text.replace(" <|tab|> ", "\t").replace("<|tab|>", "\t")
+    text = text.replace("<|indent_4|> ", "    ").replace("<|indent_4|>", "    ")
+    text = text.replace("<|indent_2|> ", "  ").replace("<|indent_2|>", "  ")
+    return text
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05fd42b359b9bf5372db4c4186e0f7c11a815886ccec35ae0607b4152989aa4b
+size 201377264
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,36 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "extra_special_tokens": [
+    "<|nl|>",
+    "<|tab|>",
+    "<|indent_4|>",
+    "<|indent_2|>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|tool|>",
+    "<|result|>",
+    "<|final|>",
+    "<|eot|>",
+    "<|synapse|>",
+    "<|route:NSR|>",
+    "<|route:TMS|>",
+    "<|route:UQM|>",
+    "<|route:WEB|>",
+    "<|route:MATH|>",
+    "<|route:IDENTITY|>",
+    "<|route:FALLBACK|>",
+    "<|web_search|>",
+    "<|math_solver|>",
+    "<|context|>",
+    "<|khazri|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}
--- a/training_state.json
+++ b/training_state.json
@@ -0,0 +1,7 @@
+{
+  "kind": "polished_non_lima_final",
+  "trained_tokens": 2060009995,
+  "global_step": 18335,
+  "base_checkpoint": "/content/drive/MyDrive/khazri_models/khazri_mini_100m_checkpoints/step_13356_tokens_1750M",
+  "saved_at": "2026-06-18 15:46:26"
+}