初始化项目，由ModelHub XC社区提供模型
Model: Abiray/Kshana-170M-Base Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
112
README.md
Normal file
112
README.md
Normal file
@@ -0,0 +1,112 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- text-generation
|
||||
- causal-lm
|
||||
- llama
|
||||
- custom-architecture
|
||||
- pytorch
|
||||
- gqa
|
||||
datasets:
|
||||
- fineweb-edu
|
||||
- cosmopedia
|
||||
---
|
||||
## 🌌 Kshana-170M Base
|
||||
|
||||
Kshana-170M-Base is a compact 170M-parameter foundational causal language model built by Abiray. Moving along the architectural lineage of its predecessor (*Sutra*), *Kshana* is trained from scratch using a highly optimized Llama-style architecture with Grouped-Query Attention (GQA) for blazing inference velocity.
|
||||
|
||||
Despite its compact size, it achieves highly competitive results on key reasoning benchmarks, making it an optimal base for downstream fine-tuning workflows or resource-constrained edge deployment.
|
||||
|
||||
*Note: As a raw base model, it requires downstream instruction tuning to perform as a conversational chat agent.*
|
||||
|
||||
## 🏆 Benchmarks
|
||||
|
||||
The base weights were evaluated head-to-head against sub-500M architectures using `lm-evaluation-harness` within an identical runtime environment. To align with open-source presentation standards, scores reflect peak performance metric selection targets (`acc` for science and single-token knowledge choice selections, `acc_norm` for length-penalized situational context completions).
|
||||
|
||||
| Benchmark | 🌌 Kshana-170M (Ours) | 🪵 SmolLM2-135M | 🌾 Nandi-Mini-150M | 📐 Pythia-160m | 🔹 OPT-125m | 🧮 Cerebras-256M | ⚙️ Pythia-410m |
|
||||
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **Parameters** | **169.9M** | 135M | 150M | 160M | 125M | 256M | 410M |
|
||||
| **SciQ** *(Sci)* | **81.90%** | 84.10% | 89.10% | 55.70% | 78.20% | 75.70% | 80.40% |
|
||||
| **PIQA** *(Logic)* | **66.81%** | 68.34% | 65.13% | 59.19% | 62.62% | 61.10% | 66.70% |
|
||||
| **ARC-Easy** *(Know)* | **57.07%** | 64.39% | 54.67% | 37.58% | 42.76% | 40.99% | 51.98% |
|
||||
| **HellaSwag** *(Ctx)*| **39.84%** | 43.17% | 37.11% | 30.49% | 31.62% | 28.60% | 40.02% |
|
||||
|
||||
## 🧠 Model Architecture
|
||||
|
||||
Kshana-170M is based on the `LlamaForCausalLM` architecture with a native Grouped-Query Attention (GQA) layout to compress hardware footprint:
|
||||
|
||||
| Parameter | Value |
|
||||
| :--- | :--- |
|
||||
| Parameters | 169,906,752 |
|
||||
| Hidden size | 576 |
|
||||
| Layers | 32 |
|
||||
| Attention heads | 9 |
|
||||
| KV heads (GQA) | 3 |
|
||||
| Head dimension | 64 |
|
||||
| Intermediate size | 1,536 |
|
||||
| Activation | SwiGLU (`silu`) |
|
||||
| Max Context | 8,192 tokens |
|
||||
| Vocabulary size | 49,152 |
|
||||
|
||||
## ⚙️ Training Configuration
|
||||
|
||||
| Parameter | Value |
|
||||
| :--- | :--- |
|
||||
| Optimizer | AdamW |
|
||||
| Learning rate | 3e-4 |
|
||||
| LR scheduler | Cosine Decay |
|
||||
| Precision | `bfloat16` / `float16` hybrid |
|
||||
|
||||
## 📚 Training Data
|
||||
|
||||
Trained on a volume of **65 Billion tokens**. The corpus characteristics include high-quality deduplicated web extracts, structured synthetic reasoning texts, and educational literature subsets (focusing on FineWeb-Edu, Wikipedia, and Cosmopedia). Data was rigorously filtered using MinHash LSH deduplication and language filtering matrices.
|
||||
|
||||
## 🎯 Operational Scope & Intended Use
|
||||
|
||||
### ✅ Targeted Applications
|
||||
* **Downstream Fine-Tuning (SFT/DPO):** Acts as a clean, lightweight base for training specialized assistants, custom chat agents, or task-specific models.
|
||||
* **Local & Edge Deployment:** Designed with Grouped-Query Attention (GQA) for efficient quantization (via `llama.cpp` / GGUF), making it ideal for low-power hardware like consumer CPUs, laptops, and mobile devices.
|
||||
* **Text Completion & Routing:** Well-suited for low-latency text continuation, basic autocomplete features, or classification tasks like routing user queries quickly before passing them to larger models.
|
||||
|
||||
### ❌ Out-of-Scope Limits
|
||||
* **Coding & Mathematics:** The model's training data consists strictly of natural language text (FineWeb-Edu and Cosmopedia). Because it was *never* exposed to structured math datasets or code repositories during training, it cannot write code scripts, debug software, or calculate mathematical formulas.
|
||||
* **Factual Knowledge Retrieval:** Trained on a strict budget of 65 Billion tokens with a sub-200M parameter boundary, the model lacks the capacity to serve as an open-domain factual encyclopedia. It will hallucinate facts if asked about niche topics without being provided reference text directly in the prompt (e.g., via RAG).
|
||||
* **Interactive Chat (Out of the box):** As a raw base model, it will naturally attempt to autocomplete text rather than hold a conversational dialogue. It requires standard instruction fine-tuning before it can be used as a traditional chatbot.
|
||||
|
||||
## 🚀 Inference & Edge Deployment
|
||||
|
||||
The model can be initialized within minutes using standard workflows via the Hugging Face `transformers` environment. Its native GQA layout makes it highly compatible with quantization layers (via `llama.cpp` / GGUF) to run on consumer CPUs or embedded devices at extreme tokens-per-second.
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "Abiray/Kshana-170M-Base"
|
||||
|
||||
# Initialize matching vocabulary tokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id, token=True)
|
||||
|
||||
# Pull weights matching verified float16 layout
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.float16,
|
||||
device_map="auto",
|
||||
token=True
|
||||
)
|
||||
|
||||
prompt = "The basic physical principle behind gravitational collapse is"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=64,
|
||||
temperature=0.6,
|
||||
top_p=0.85,
|
||||
do_sample=True,
|
||||
pad_token_id=tokenizer.eos_token_id
|
||||
)
|
||||
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
20
chat_template.jinja
Normal file
20
chat_template.jinja
Normal file
@@ -0,0 +1,20 @@
|
||||
# Instructions
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{ messages[0].content }}
|
||||
{%- set context_messages = messages[1:-1] %}
|
||||
{%- set last_user_message = messages[-1] %}
|
||||
{%- else %}
|
||||
{%- set context_messages = messages[:-1] %}
|
||||
{%- set last_user_message = messages[-1] %}
|
||||
{%- endif %}
|
||||
|
||||
# Context
|
||||
{%- for message in context_messages %}
|
||||
{{ message.role }}: {{ message.content }}
|
||||
{%- endfor %}
|
||||
|
||||
# Prompt
|
||||
user: {{ last_user_message.content }}
|
||||
|
||||
# Response
|
||||
assistant:
|
||||
34
config.json
Normal file
34
config.json
Normal file
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 0,
|
||||
"dtype": "float16",
|
||||
"eos_token_id": 0,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 576,
|
||||
"initializer_range": 0.041666666666666664,
|
||||
"intermediate_size": 1536,
|
||||
"is_llama_config": true,
|
||||
"max_position_embeddings": 8192,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 9,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 3,
|
||||
"pad_token_id": 0,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_interleaved": false,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 100000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "5.10.2",
|
||||
"use_cache": false,
|
||||
"vocab_size": 49152
|
||||
}
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": [
|
||||
0
|
||||
],
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "5.10.2",
|
||||
"use_cache": true
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3ab9b220bed728275e37848aa1415349ff2ca9a874d2b644f7595c2a42d2f02f
|
||||
size 339845984
|
||||
244965
tokenizer.json
Normal file
244965
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
15
tokenizer_config.json
Normal file
15
tokenizer_config.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<|endoftext|>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"errors": "replace",
|
||||
"is_local": true,
|
||||
"local_files_only": false,
|
||||
"model_max_length": 8192,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|endoftext|>",
|
||||
"vocab_size": 49152
|
||||
}
|
||||
Reference in New Issue
Block a user