初始化项目，由ModelHub XC社区提供模型

Model: vedantjadhav701/SparkAI-47m-llama-instruct
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,118 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- llama
+- causal-lm
+- text-generation
+- sft
+- chat
+- pytorch
+- fineweb-edu
+- cosmopedia
+pipeline_tag: text-generation
+datasets:
+- HuggingFaceFW/fineweb-edu
+- HuggingFaceTB/cosmopedia-v2
+base_model: vedantjadhav701/SparkAI-47m-llama-10b-token
+---
+
+# SparkAI-47M-Llama-Instruct
+
+[![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-SparkAI--47m--llama--instruct-blue)](https://huggingface.co/vedantjadhav701/SparkAI-47m-llama-instruct)
+[![GitHub Repository](https://img.shields.io/badge/GitHub-SparkAI-black?logo=github)](https://github.com/VedantJadhav701/SparkAI)
+
+Instruction-tuned checkpoint of **SparkAI-47M-Llama** (~48M parameter decoder-only transformer), fine-tuned for chat and instruction following.
+
+> 🌟 **Key Highlights & Unique Features**
+> * ⚡ **Ultra-Low Memory Footprint (~95.4 MB):** Fits in under 100MB of RAM, making it suitable for edge devices, mobile apps, WebGPU, and microcontrollers.
+> * 🏋️ **Data-Saturated Pretraining (10 Billion Tokens):** Pretrained on 10B tokens (210 tokens/param) of high-quality FineWeb-Edu + Cosmopedia-v2 text, providing an empirical benchmark on capacity saturation for sub-50M models.
+> * 🏗️ **Modern LLaMA 3 Architecture:** Built with Grouped Query Attention (GQA), SwiGLU activations, RoPE positional encodings, RMSNorm pre-normalization, and tied embeddings.
+> * 💬 **Full ChatML SFT Alignment:** Fine-tuned with ChatML `<|im_start|>` instruction formatting and template support (`chat_template.jinja`).
+
+---
+
+## 📐 Architecture
+* **Parameters:** ~48M (~47.4M non-embedding / tied embeddings)
+* **Layers:** 8
+* **Hidden Size:** 512
+* **Attention Heads:** 8 query heads, 2 key/value heads (Grouped Query Attention - GQA)
+* **MLP:** SwiGLU (Intermediate size: 1408)
+* **Positional Encoding:** RoPE (Rotary Position Embeddings)
+* **Normalization:** RMSNorm (Pre-normalization)
+* **Embeddings:** Tied embeddings, no bias terms
+* **Vocabulary Size:** 49,152 (SmolLM2 tokenizer with Chat Template)
+* **Sequence Length:** 1024
+
+---
+
+## 🏋️ Training & Fine-Tuning Details
+* **Base Checkpoint:** [`vedantjadhav701/SparkAI-47m-llama-10b-token`](https://huggingface.co/vedantjadhav701/SparkAI-47m-llama-10b-token)
+* **Pretraining Data:** FineWeb-Edu (`sample-100BT`) + Cosmopedia-v2 (**85% / 15%** mix, 10.00B tokens)
+* **Optimizer:** AdamW with Cosine LR schedule + warmup
+* **Hardware:** NVIDIA A100 80GB PCIe
+* **Final Eval Perplexity:** 31.46
+
+### 📈 Pretraining Progression
+
+| Tokens | Perplexity |
+| :--- | :--- |
+| **630M** | 43.49 |
+| **3.77B** | 31.30 |
+| **7.00B** | — |
+| **10.00B** | 31.46 |
+
+> **Note on Saturation:** Perplexity plateaued between 3.77B and 10.00B tokens despite continued training, indicating the model has saturated its representational capacity at this size.
+
+---
+
+## 📊 Comparison in Sub-50M Parameter Landscape
+
+| Feature | Typical Sub-50M Models | **SparkAI-47M-Llama / Instruct** |
+| :--- | :--- | :--- |
+| **Token Budget** | ~1B – 2B tokens | **10.00 Billion Tokens** (210 tokens/param) |
+| **Data Quality** | Raw web text / C4 | **FineWeb-Edu (85%) + Cosmopedia-v2 (15%)** |
+| **Architecture** | Basic MHA / GPT-2 style | **Modern LLaMA 3 (GQA, SwiGLU, RoPE, RMSNorm)** |
+| **Model Size** | ~100MB – 200MB | **~95.4 MB (`model.safetensors`)** |
+| **SFT Alignment** | Rare / None | **Instruction-tuned with ChatML (`chat_template.jinja`)** |
+| **Benchmarking** | Few metrics | **Empirical capacity saturation documented at 10B tokens** |
+
+---
+
+## 💻 Usage with Hugging Face `transformers`
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "vedantjadhav701/SparkAI-47m-llama-instruct"
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForCausalLM.from_pretrained(repo_id)
+
+messages = [
+    {"role": "user", "content": "What is a computer program?"}
+]
+
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.6)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+---
+
+## 🚀 Quickstart & Local Gradio UI
+
+### Installation
+```bash
+pip install -r requirements.txt
+```
+
+### Run Gradio App
+```bash
+python app.py
+```
+Open `http://127.0.0.1:7860` in your web browser.

chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>
+'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant
+'}}{% endif %}

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1408,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 2,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 49152
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4d34a4449d631741802175110de7a387796d183ff8b44554f934578da3051cb
+size 95445968

tokenizer_config.json

@@ -0,0 +1,19 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}