初始化项目，由ModelHub XC社区提供模型
Model: Yusiko/khazri-2-mini Source: Original Platform
This commit is contained in:
ModelHub XC
143
README.md
Normal file
143
README.md
Normal file
@@ -0,0 +1,143 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
datasets:
|
||||
- HuggingFaceFW/fineweb
|
||||
- bigcode/the-stack-dedup
|
||||
- GAIR/lima
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: text-classification
|
||||
tags:
|
||||
- khazri
|
||||
- softyu
|
||||
- ai
|
||||
- llm
|
||||
- azerbaijani-national-llm
|
||||
- azerbaijan
|
||||
---
|
||||
<p align="center">
|
||||
<img src="./assets/khazri-2-mini-banner.png" alt="Khazri 2 Mini — compact open-weight language model" width="100%">
|
||||
</p>
|
||||
|
||||
# Khazri 2 Mini — 100M
|
||||
|
||||
**Khazri 2 Mini** is a compact, open-weight decoder-only language model in the Khazri family. It combines a modern LLaMA-style Transformer with a custom Byte-Level BPE tokenizer and a 2B-token English training corpus.
|
||||
|
||||
[Hugging Face](https://huggingface.co/Yusiko/khazri-2-mini) · [Khazri](https://khazri.dev) · [Contact](mailto:contact@khazri.dev)
|
||||
|
||||
## At a glance
|
||||
|
||||
| Item | Detail |
|
||||
| --- | --- |
|
||||
| Model | Khazri 2 Mini |
|
||||
| Parameters | **100.68M** |
|
||||
| Status | Open weights on Hugging Face |
|
||||
| Architecture | LLaMA-style, decoder-only Transformer |
|
||||
| Training precision | bf16 |
|
||||
| Context configured for training | 1,024 tokens |
|
||||
| Vocabulary | 32,768 tokens |
|
||||
| Tokenizer | Custom Byte-Level BPE |
|
||||
| Attention | Grouped-Query Attention: 12 query heads / 4 KV heads |
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Configuration |
|
||||
| --- | --- |
|
||||
| Hidden size | 768 |
|
||||
| Transformer layers | 12 |
|
||||
| Attention heads | 12 |
|
||||
| Key/value heads | 4 |
|
||||
| MLP intermediate size | 2,048 |
|
||||
| Positional encoding | RoPE |
|
||||
| Normalization | RMSNorm |
|
||||
| MLP activation | SwiGLU / SiLU |
|
||||
| Attention backend | FlashAttention-2 where available; PyTorch SDPA fallback |
|
||||
| Embeddings | Tied input/output embeddings |
|
||||
|
||||
## Training data
|
||||
|
||||
Khazri 2 Mini is trained on a custom, pretokenized **English-only** corpus with a target size of **2,000,000,000 tokens**. The corpus is packed into **1,953,125 sequences** of 1,024 tokens and stored in Arrow shards with source identifiers.
|
||||
|
||||
The documented token budget is:
|
||||
|
||||
| Source | Token budget | Share | Role |
|
||||
| --- | ---: | ---: | --- |
|
||||
| [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) | 850M | 42.5% | General English educational and synthetic-text coverage |
|
||||
| [The Stack v2 Dedup](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) | 450M | 22.5% | Code from Python, JavaScript, TypeScript, Java, C++, C, Go and Rust |
|
||||
| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) | 300M | 15.0% | Simple narrative language |
|
||||
| [LIMA](https://huggingface.co/datasets/GAIR/lima) | 50M | 2.5% | Instruction and conversation examples |
|
||||
| SYNAPSE synthetic instruction data | 350M | 17.5% | Arithmetic, context, abstention, web-needed, identity, symbolic-math and general-assistant routes |
|
||||
|
||||
The listed values are the documented source-token budget. The release manifest should be used for the final source counts of a particular weight revision.
|
||||
|
||||
### Data processing and safeguards
|
||||
|
||||
- Only English text is retained for this corpus. Short records are excluded, very long records are capped, and language/character checks are applied before tokenization.
|
||||
- The code portion is limited to the eight languages listed above.
|
||||
- The training mix combines general English, narrative, code, instruction and route-aware synthetic material. This preserves general capabilities while teaching specialized SYNAPSE behaviours.
|
||||
- Checkpoint evaluation should cover general Q&A, code, arithmetic, context extraction, abstention and current-information requests. A route-specific gain should not be accepted if it degrades general behaviour.
|
||||
- Original source datasets remain subject to their own terms and licences. Consult their source pages and the model repository licence before use.
|
||||
|
||||
## Tokenizer
|
||||
|
||||
Khazri 2 Mini uses a 32,768-token custom Byte-Level BPE tokenizer. It reserves structural whitespace, chat and SYNAPSE route tokens as single tokens, preserves indentation for code, and uses single-digit splitting to make arithmetic strings more explicit to the model.
|
||||
|
||||
## Installation
|
||||
|
||||
~~~bash
|
||||
pip install -U torch transformers accelerate safetensors
|
||||
~~~
|
||||
|
||||
## Quick start
|
||||
|
||||
~~~python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
MODEL_ID = "Yusiko/khazri-2-mini"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL_ID,
|
||||
torch_dtype="auto",
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
prompt = "Write a concise explanation of a small language model."
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.inference_mode():
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=160,
|
||||
do_sample=False,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||||
~~~
|
||||
|
||||
## Khazri 2 Preview comparison
|
||||
|
||||
The following table reports the project-provided compact-model comparison for **Khazri 2 Preview**, not Khazri 2 Mini. Higher is better for every listed task.
|
||||
|
||||
| Model | Parameters | Context extraction | Mixed speed/proxy | Arithmetic | Word problems | Abstention |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Khazri 2 Preview** | ~250M | **100%** | **62%** | **99%** | **99%** | **97.4%** |
|
||||
| Gemma 3 | 270M | 100% | 36% | 0% | 0% | 18% |
|
||||
| Qwen 2.5 | 0.5B | 89% | 45% | 14% | 28% | 46% |
|
||||
| Pythia | 160M | 22% | 10% | 0% | 2% | 1% |
|
||||
|
||||
These are internal preview results on selected compact-model tests. They are not independently audited and should not be used to make claims about Khazri 2 Mini. Publish prompts, model revisions, scoring rules, hardware and complete evaluation assets with any future benchmark announcement.
|
||||
|
||||
## Responsible use
|
||||
|
||||
Khazri 2 Mini can produce incorrect, incomplete or biased outputs. Evaluate it on your own task, verify material claims and keep a human in the loop for consequential decisions. Do not rely on it as the sole basis for legal, medical, financial, hiring, safety or other high-impact decisions.
|
||||
|
||||
## Roadmap
|
||||
|
||||
Khazri 2 Mini is part of the second Khazri generation. The next planned stage is **Khazri 3**: a larger parameter scale and stronger results.
|
||||
|
||||
## Contact
|
||||
|
||||
For research, integration or partnership inquiries, visit [khazri.dev](https://khazri.dev) or email [contact@khazri.dev](mailto:contact@khazri.dev).
|
||||
Reference in New Issue
Block a user