初始化项目，由ModelHub XC社区提供模型

Model: Abiray/Kshana-170M-Base
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,112 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- text-generation
+- causal-lm
+- llama
+- custom-architecture
+- pytorch
+- gqa
+datasets:
+- fineweb-edu
+- cosmopedia
+---
+## 🌌 Kshana-170M Base
+
+Kshana-170M-Base is a compact 170M-parameter foundational causal language model built by Abiray. Moving along the architectural lineage of its predecessor (*Sutra*), *Kshana* is trained from scratch using a highly optimized Llama-style architecture with Grouped-Query Attention (GQA) for blazing inference velocity. 
+
+Despite its compact size, it achieves highly competitive results on key reasoning benchmarks, making it an optimal base for downstream fine-tuning workflows or resource-constrained edge deployment.
+
+*Note: As a raw base model, it requires downstream instruction tuning to perform as a conversational chat agent.*
+
+## 🏆 Benchmarks
+
+The base weights were evaluated head-to-head against sub-500M architectures using `lm-evaluation-harness` within an identical runtime environment. To align with open-source presentation standards, scores reflect peak performance metric selection targets (`acc` for science and single-token knowledge choice selections, `acc_norm` for length-penalized situational context completions).
+
+| Benchmark | 🌌 Kshana-170M (Ours) | 🪵 SmolLM2-135M | 🌾 Nandi-Mini-150M | 📐 Pythia-160m | 🔹 OPT-125m | 🧮 Cerebras-256M | ⚙️ Pythia-410m |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Parameters** | **169.9M** | 135M | 150M | 160M | 125M | 256M | 410M |
+| **SciQ** *(Sci)* | **81.90%** | 84.10% | 89.10% | 55.70% | 78.20% | 75.70% | 80.40% |
+| **PIQA** *(Logic)* | **66.81%** | 68.34% | 65.13% | 59.19% | 62.62% | 61.10% | 66.70% |
+| **ARC-Easy** *(Know)* | **57.07%** | 64.39% | 54.67% | 37.58% | 42.76% | 40.99% | 51.98% |
+| **HellaSwag** *(Ctx)*| **39.84%** | 43.17% | 37.11% | 30.49% | 31.62% | 28.60% | 40.02% |
+
+## 🧠 Model Architecture
+
+Kshana-170M is based on the `LlamaForCausalLM` architecture with a native Grouped-Query Attention (GQA) layout to compress hardware footprint:
+
+| Parameter | Value |
+| :--- | :--- |
+| Parameters | 169,906,752 |
+| Hidden size | 576 |
+| Layers | 32 |
+| Attention heads | 9 |
+| KV heads (GQA) | 3 |
+| Head dimension | 64 |
+| Intermediate size | 1,536 |
+| Activation | SwiGLU (`silu`) |
+| Max Context | 8,192 tokens |
+| Vocabulary size | 49,152 |
+
+## ⚙️ Training Configuration
+
+| Parameter | Value |
+| :--- | :--- |
+| Optimizer | AdamW |
+| Learning rate | 3e-4 |
+| LR scheduler | Cosine Decay |
+| Precision | `bfloat16` / `float16` hybrid |
+
+## 📚 Training Data
+
+Trained on a volume of **65 Billion tokens**. The corpus characteristics include high-quality deduplicated web extracts, structured synthetic reasoning texts, and educational literature subsets (focusing on FineWeb-Edu, Wikipedia, and Cosmopedia). Data was rigorously filtered using MinHash LSH deduplication and language filtering matrices.
+
+## 🎯 Operational Scope & Intended Use
+
+### ✅ Targeted Applications
+* **Downstream Fine-Tuning (SFT/DPO):** Acts as a clean, lightweight base for training specialized assistants, custom chat agents, or task-specific models.
+* **Local & Edge Deployment:** Designed with Grouped-Query Attention (GQA) for efficient quantization (via `llama.cpp` / GGUF), making it ideal for low-power hardware like consumer CPUs, laptops, and mobile devices.
+* **Text Completion & Routing:** Well-suited for low-latency text continuation, basic autocomplete features, or classification tasks like routing user queries quickly before passing them to larger models.
+
+### ❌ Out-of-Scope Limits
+* **Coding & Mathematics:** The model's training data consists strictly of natural language text (FineWeb-Edu and Cosmopedia). Because it was *never* exposed to structured math datasets or code repositories during training, it cannot write code scripts, debug software, or calculate mathematical formulas.
+* **Factual Knowledge Retrieval:** Trained on a strict budget of 65 Billion tokens with a sub-200M parameter boundary, the model lacks the capacity to serve as an open-domain factual encyclopedia. It will hallucinate facts if asked about niche topics without being provided reference text directly in the prompt (e.g., via RAG).
+* **Interactive Chat (Out of the box):** As a raw base model, it will naturally attempt to autocomplete text rather than hold a conversational dialogue. It requires standard instruction fine-tuning before it can be used as a traditional chatbot.
+
+## 🚀 Inference & Edge Deployment
+
+The model can be initialized within minutes using standard workflows via the Hugging Face `transformers` environment. Its native GQA layout makes it highly compatible with quantization layers (via `llama.cpp` / GGUF) to run on consumer CPUs or embedded devices at extreme tokens-per-second.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Abiray/Kshana-170M-Base"
+
+# Initialize matching vocabulary tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_id, token=True)
+
+# Pull weights matching verified float16 layout
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+    token=True
+)
+
+prompt = "The basic physical principle behind gravitational collapse is"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs, 
+        max_new_tokens=64,
+        temperature=0.6,
+        top_p=0.85,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))

chat_template.jinja

@@ -0,0 +1,20 @@
+# Instructions
+{%- if messages[0].role == 'system' %}
+{{ messages[0].content }}
+{%- set context_messages = messages[1:-1] %}
+{%- set last_user_message = messages[-1] %}
+{%- else %}
+{%- set context_messages = messages[:-1] %}
+{%- set last_user_message = messages[-1] %}
+{%- endif %}
+
+# Context
+{%- for message in context_messages %}
+{{ message.role }}: {{ message.content }}
+{%- endfor %}
+
+# Prompt
+user: {{ last_user_message.content }}
+
+# Response
+assistant:

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float16",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 576,
+  "initializer_range": 0.041666666666666664,
+  "intermediate_size": 1536,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 9,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 3,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.10.2",
+  "use_cache": false,
+  "vocab_size": 49152
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    0
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.10.2",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ab9b220bed728275e37848aa1415349ff2ca9a874d2b644f7595c2a42d2f02f
+size 339845984

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}