初始化项目，由ModelHub XC社区提供模型
Model: nareshmeena12/fluxion-370m-instruct Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
364
README.md
Normal file
364
README.md
Normal file
@@ -0,0 +1,364 @@
|
|||||||
|
---
|
||||||
|
library_name: transformers
|
||||||
|
tags:
|
||||||
|
- code
|
||||||
|
- math
|
||||||
|
- text-generation-inference
|
||||||
|
- slm
|
||||||
|
- instructmodel
|
||||||
|
license: apache-2.0
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
---
|
||||||
|
|
||||||
|
# ⚡ Fluxion-370M-Instruct
|
||||||
|
|
||||||
|
### *Trained on 0.227T tokens. Competing with models trained on 28T.*
|
||||||
|
|
||||||
|
<br/>
|
||||||
|
|
||||||
|
[](<!-- ADD -->)
|
||||||
|
[](https://huggingface.co/nareshmeena12/fluxion-370m-instruct)
|
||||||
|
[](<!-- ADD -->)
|
||||||
|
[](https://www.apache.org/licenses/LICENSE-2.0)
|
||||||
|
|
||||||
|
<div align="center">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What is Fluxion?
|
||||||
|
|
||||||
|
**Fluxion-370M-Instruct** is a 370M-parameter instruction-tuned language model built from scratch, focused on **mathematical reasoning and code generation**. Its core design philosophy is simple: *do more with less*.
|
||||||
|
|
||||||
|
Trained on just **227B tokens** — a fraction of what comparable models consume — Fluxion achieves competitive performance on math and coding benchmarks against models that have seen **18–28× more data**. On token efficiency (MATH-500 score per trillion training tokens), Fluxion ranks **1st** across all models in its class by a dominant margin.
|
||||||
|
|
||||||
|
The model went through three training stages: large-scale pretraining, supervised fine-tuning for instruction following, and GRPO reinforcement learning specifically targeting mathematical reasoning with rule-based rewards.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚡ Token Efficiency — The Core Story
|
||||||
|
|
||||||
|
| Model | Training Tokens | MATH-500 | **MATH-500 / Trillion Tokens** |
|
||||||
|
|:---|:---:|:---:|:---:|
|
||||||
|
| **Fluxion-370M** | **0.227T** | **17.40%** | **🥇 76.7×** |
|
||||||
|
| MobileLLM-R1-360M | 4.2T | 25.60% | 6.1× |
|
||||||
|
| Gemma3-270M | 6T | 8.20% | 1.4× |
|
||||||
|
| Granite-350M | 11T | 19.20% | 1.7× |
|
||||||
|
| Qwen2.5-0.5B | 18T | 29.80% | 1.7× |
|
||||||
|
| LFM2.5-350M | 28T | 13.40% | 0.5× |
|
||||||
|
| SmolLM2-360M | 4T | 3.40% | 0.9× |
|
||||||
|
|
||||||
|
**Fluxion delivers 76.7× higher MATH-500 score per trillion training tokens than any other model in this comparison.** This is a structural advantage from architecture choices, custom tokenization, and a high-signal data pipeline — not a measurement artifact.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏗️ Architecture
|
||||||
|
|
||||||
|
Fluxion is a clean, modern decoder-only transformer. No hybrid layers, no custom operators — just well-chosen standard components assembled deliberately.
|
||||||
|
|
||||||
|
```
|
||||||
|
FluxionForCausalLM
|
||||||
|
├── Embedding (vocab_size=50,000 · hidden_size=1024)
|
||||||
|
├── 26 × DecoderBlock
|
||||||
|
│ ├── RMSNorm (ε = 1e-6)
|
||||||
|
│ ├── GroupedQueryAttention
|
||||||
|
│ │ ├── 16 Query heads · 8 KV heads (GQA 2:1)
|
||||||
|
│ │ └── RoPE (θ = 500,000) + NTK scaling
|
||||||
|
│ ├── RMSNorm (ε = 1e-6)
|
||||||
|
│ └── SwiGLU FFN (hidden → 3072 → hidden)
|
||||||
|
└── RMSNorm → LM Head (weight-tied to embedding)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hyperparameters
|
||||||
|
|
||||||
|
| Parameter | Value |
|
||||||
|
|:---|:---:|
|
||||||
|
| **Total Parameters** | ~370M |
|
||||||
|
| **Layers** | 26 |
|
||||||
|
| **Hidden Size** | 1024 |
|
||||||
|
| **FFN Intermediate Size** | 3072 |
|
||||||
|
| **Attention Heads (Q)** | 16 |
|
||||||
|
| **KV Heads (GQA)** | 8 |
|
||||||
|
| **GQA Group Ratio** | 2 : 1 |
|
||||||
|
| **Head Dimension** | 64 |
|
||||||
|
| **Context Length** | 4096 |
|
||||||
|
| **Vocabulary Size** | 50,000 |
|
||||||
|
| **RoPE Base θ** | 500,000 |
|
||||||
|
| **RoPE Scaling** | NTK |
|
||||||
|
| **Normalization** | RMSNorm (ε = 1e-6) |
|
||||||
|
| **Activation** | SwiGLU |
|
||||||
|
| **Attention** | Flash Attention |
|
||||||
|
| **Weight Tying** | Yes (embed ↔ lm\_head) |
|
||||||
|
| **Attention Bias** | No |
|
||||||
|
| **FFN Bias** | No |
|
||||||
|
| **dtype** | bfloat16 |
|
||||||
|
|
||||||
|
### Design Choices & Rationale
|
||||||
|
|
||||||
|
**Grouped Query Attention (16Q / 8KV):** Halves KV cache memory vs standard MHA with no meaningful quality loss at this scale. Larger effective batch sizes at both training and inference time.
|
||||||
|
|
||||||
|
**RoPE with θ = 500,000 + NTK scaling:** High base frequency improves length generalisation beyond the training window. NTK scaling further stabilises attention patterns at longer sequences without fine-tuning.
|
||||||
|
|
||||||
|
**SwiGLU activation:** Consistently stronger than GeLU/ReLU on reasoning tasks. The gated structure gives the FFN richer expressiveness per parameter.
|
||||||
|
|
||||||
|
**Custom BPE tokenizer (50k vocab):** Trained from scratch on 100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. A smaller, domain-tuned vocabulary means fewer tokens per math expression and code snippet, directly improving effective context utilisation and reducing sequence lengths vs generic 128k tokenizers.
|
||||||
|
|
||||||
|
**Weight tying (embed ↔ lm\_head):** Saves ~50M parameters, keeps the model compact, and forces the input and output representation spaces to align.
|
||||||
|
|
||||||
|
**Flash Attention:** Full Flash Attention used throughout training for memory efficiency and throughput. TF32 enabled for matrix multiplications.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Benchmarks
|
||||||
|
|
||||||
|
All results are **0-shot** unless noted. Evaluation is strict.
|
||||||
|
|
||||||
|
### Math & Code
|
||||||
|
|
||||||
|
| Model | Tokens | GSM8K | MATH-500 | HumanEval | MBPP | **Avg** |
|
||||||
|
|:---|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||||
|
| Qwen2.5-0.5B | 18T | 43.44% | 29.80% | 30.49% | 38.91% | 35.66% |
|
||||||
|
| Granite-350M | 11T | 31.69% | 19.20% | 29.88% | 42.02% | 30.70% |
|
||||||
|
| LFM2.5-350M | 28T | 32.68% | 13.40% | 10.37% | 5.45% | 15.48% |
|
||||||
|
| MobileLLM-R1-360M | 4.2T | 23.81% | 25.60% | 19.51% | 17.51% | 21.61% |
|
||||||
|
| **Fluxion-370M** | **0.227T** | **29.57%** | **17.40%** | **16.46%** | **20.23%** | **20.92%** |
|
||||||
|
| SmolLM2-360M | 4T | 8.87% | 3.40% | 17.68% | 35.02% | 16.24% |
|
||||||
|
| Gemma3-270M | 6T | 7.58% | 8.20% | 11.59% | 19.07% | 11.61% |
|
||||||
|
|
||||||
|
### General Reasoning
|
||||||
|
|
||||||
|
| Model | ARC | PIQA | OBQA | CSQA | MMLU | Wino | IFEval | **Avg** |
|
||||||
|
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||||
|
| Qwen2.5-0.5B | 50.9% | 70.1% | 36.2% | 42.3% | 45.9% | 51.1% | 74.1% | 52.9% |
|
||||||
|
| Granite-350M | 51.8% | 68.9% | 37.8% | 40.5% | 33.8% | 53.8% | 83.2% | 52.8% |
|
||||||
|
| LFM2.5-350M | 49.4% | 67.0% | 31.6% | 33.4% | 42.2% | 52.0% | 88.3% | 52.0% |
|
||||||
|
| SmolLM2-360M | 45.9% | 71.5% | 35.2% | 29.0% | 24.9% | 54.8% | 77.5% | 48.4% |
|
||||||
|
| Gemma3-270M | 43.7% | 66.6% | 30.6% | 36.9% | 24.7% | 49.7% | 71.5% | 46.2% |
|
||||||
|
| MobileLLM-R1-360M | 39.1% | 58.6% | 26.2% | 27.6% | 26.7% | 50.8% | 71.2% | 42.9% |
|
||||||
|
| **Fluxion-370M** | **36.3%** | **61.6%** | **29.4%** | **27.3%** | **25.3%** | **51.9%** | **66.9%** | **42.7%** |
|
||||||
|
|
||||||
|
### Rankings Summary
|
||||||
|
|
||||||
|
| Category | Score | Rank |
|
||||||
|
|:---|:---:|:---:|
|
||||||
|
| Math Average (GSM8K + MATH-500) | 23.49% | **4th** |
|
||||||
|
| Code Average (HumanEval + MBPP) | 18.35% | **5th** |
|
||||||
|
| Math + Code Combined | 20.92% | **4th** |
|
||||||
|
| General Average | 39.19% | 7th |
|
||||||
|
| **Token Efficiency (MATH-500 / T tokens)** | **76.7×** | **🥇 1st** |
|
||||||
|
|
||||||
|
> Fluxion's design target is math and code under tight data constraints. On general benchmarks like IFEval and MMLU it trails models trained on significantly more diverse data — this is expected and intentional. On token efficiency it has no close competitor.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 💬 Chat Template & Tokenizer
|
||||||
|
|
||||||
|
Fluxion-Instruct uses the **ChatML** format:
|
||||||
|
|
||||||
|
```
|
||||||
|
<|im_start|>system
|
||||||
|
You are a helpful assistant.<|im_end|>
|
||||||
|
<|im_start|>user
|
||||||
|
{your prompt here}<|im_end|>
|
||||||
|
<|im_start|>assistant
|
||||||
|
```
|
||||||
|
|
||||||
|
### Special Tokens
|
||||||
|
|
||||||
|
| Token | ID | Role |
|
||||||
|
|:---|:---:|:---|
|
||||||
|
| `<\|im_start\|>` | 100278 | Turn start marker |
|
||||||
|
| `<\|im_end\|>` | 100279 | Turn end marker |
|
||||||
|
| `<\|pad\|>` | — | Padding token |
|
||||||
|
|
||||||
|
### Stopping Token IDs
|
||||||
|
|
||||||
|
Always pass `eos_token_id=[1, 4]` during generation. Without these, the model may continue generating beyond the intended response boundary:
|
||||||
|
|
||||||
|
```python
|
||||||
|
output = model.generate(
|
||||||
|
**inputs,
|
||||||
|
eos_token_id=[1, 4],
|
||||||
|
...
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Token ID `1` is the primary EOS token and token ID `4` is the `<|im_end|>` marker. Including both ensures clean turn termination in all inference scenarios.
|
||||||
|
|
||||||
|
### Tokenizer
|
||||||
|
|
||||||
|
Fluxion uses a **custom BPE tokenizer** trained from scratch on ~100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. With a **50,000 token vocabulary**, it is significantly more compact than general-purpose tokenizers (32k–128k). This means:
|
||||||
|
|
||||||
|
- Fewer tokens per math expression and code snippet
|
||||||
|
- Better effective context utilisation within the 4096 token window
|
||||||
|
- Faster embedding lookups and a smaller lm\_head projection
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
"nareshmeena12/fluxion-370m-instruct",
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
device_map="auto",
|
||||||
|
)
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(
|
||||||
|
"nareshmeena12/fluxion-370m-instruct"
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": "You are a helpful assistant that solves math problems step by step."},
|
||||||
|
{"role": "user", "content": "What is the sum of the first 100 natural numbers?"}
|
||||||
|
]
|
||||||
|
|
||||||
|
text = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=False,
|
||||||
|
add_generation_prompt=True
|
||||||
|
)
|
||||||
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
output = model.generate(
|
||||||
|
**inputs,
|
||||||
|
max_new_tokens=512,
|
||||||
|
eos_token_id=[1, 4],
|
||||||
|
do_sample=True,
|
||||||
|
top_p=0.9,
|
||||||
|
)
|
||||||
|
|
||||||
|
response = tokenizer.decode(
|
||||||
|
output[0, inputs["input_ids"].shape[1]:],
|
||||||
|
skip_special_tokens=True
|
||||||
|
)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
> **Note:** Always include `eos_token_id=[1, 4]` in all `generate()` calls regardless of quantization mode.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🗂️ Training
|
||||||
|
|
||||||
|
Fluxion was trained in three stages.
|
||||||
|
|
||||||
|
### Stage 1 — Pretraining
|
||||||
|
|
||||||
|
Large-scale unsupervised pretraining on a curated mix of math, code, and English text.
|
||||||
|
|
||||||
|
| Parameter | Value |
|
||||||
|
|:---|:---:|
|
||||||
|
| **Total Tokens** | ~227B (0.227T) |
|
||||||
|
| **Steps** | 264,000 |
|
||||||
|
| **Batch Size** | 10 sequences |
|
||||||
|
| **Gradient Accumulation** | 21 steps |
|
||||||
|
| **Effective Batch** | ~860K tokens/step |
|
||||||
|
| **Learning Rate** | 3e-4 |
|
||||||
|
| **Min Learning Rate** | 9e-5 |
|
||||||
|
| **LR Schedule** | Cosine decay |
|
||||||
|
| **Warmup Steps** | 2,000 |
|
||||||
|
| **Sequence Length** | 4,096 |
|
||||||
|
| **Optimizer** | AdamW |
|
||||||
|
| **Precision** | bfloat16 + TF32 |
|
||||||
|
| **Attention** | Flash Attention |
|
||||||
|
| **Positional Encoding** | RoPE (θ=500k) + NTK scaling |
|
||||||
|
|
||||||
|
**Pretraining data:** *(to be added)*
|
||||||
|
|
||||||
|
### Stage 2 — Supervised Fine-Tuning (SFT)
|
||||||
|
|
||||||
|
Instruction tuning using ChatML format to teach the model to follow user instructions across math, code, and general tasks.
|
||||||
|
|
||||||
|
| Parameter | Value |
|
||||||
|
|:---|:---:|
|
||||||
|
| **Format** | ChatML |
|
||||||
|
| **Data** | *(to be added)* |
|
||||||
|
|
||||||
|
### Stage 3 — GRPO (Math Reasoning)
|
||||||
|
|
||||||
|
Group Relative Policy Optimisation applied specifically to strengthen mathematical reasoning.
|
||||||
|
|
||||||
|
| Parameter | Value |
|
||||||
|
|:---|:---:|
|
||||||
|
| **Algorithm** | GRPO |
|
||||||
|
| **Reward Signal** | Rule-based answer correctness |
|
||||||
|
| **Target Domain** | Mathematics |
|
||||||
|
| **Data** | *(to be added)* |
|
||||||
|
|
||||||
|
Rule-based rewards verify final answer correctness against ground truth — no LLM judge, no model-in-the-loop. This keeps the reward signal clean, fast, and free from reward hacking through verbosity or style.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Intended Use
|
||||||
|
|
||||||
|
**Fluxion-370M-Instruct is designed for:**
|
||||||
|
- Mathematical problem solving and step-by-step reasoning
|
||||||
|
- Code generation and explanation (Python, general purpose)
|
||||||
|
- Instruction-following tasks in math and code domains
|
||||||
|
- Fine-tuning base for domain-specific math/code applications
|
||||||
|
- Efficient deployment — capable performance on modest hardware
|
||||||
|
- Research into token-efficient pretraining and RL for reasoning
|
||||||
|
|
||||||
|
**Fluxion-370M-Instruct is NOT designed for:**
|
||||||
|
- Knowledge-intensive QA requiring broad world knowledge
|
||||||
|
- Long-document tasks beyond 4096 tokens
|
||||||
|
- Multilingual use cases
|
||||||
|
- Safety-critical production deployments without additional alignment
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ Limitations
|
||||||
|
|
||||||
|
- **Narrow training data scope.** 0.227T tokens is intentionally small and domain-focused. The model has limited world knowledge outside math, code, and general English.
|
||||||
|
- **Context length is 4096 tokens.** Not suitable for long-document tasks without chunking strategies.
|
||||||
|
- **General benchmarks lag.** IFEval (66.9%) and MMLU (25.3%) reflect that the model was not optimised for instruction following diversity or broad knowledge recall — this is a deliberate trade-off.
|
||||||
|
- **Custom tokenizer.** The 50k BPE vocabulary is domain-tuned. Rare tokens, non-English text, and highly specialised symbols outside the training distribution may tokenise suboptimally.
|
||||||
|
- **Small scale.** At 370M parameters, the model is outperformed by larger models on complex multi-step reasoning chains requiring deep world knowledge.
|
||||||
|
- **Always set `eos_token_id=[1, 4]`.** Without explicit stopping token IDs the model may continue generating past the intended response boundary.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📰 News
|
||||||
|
|
||||||
|
- `<!-- ADD DATE -->` — Fluxion-370M-Instruct released on Hugging Face
|
||||||
|
- `<!-- ADD DATE -->` — Training code released on GitHub
|
||||||
|
- `<!-- ADD DATE -->` — Technical report available on arXiv
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📄 Citation
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@misc{fluxion2025,
|
||||||
|
title = {Fluxion-370M: Token-Efficient Pretraining and Reinforcement Learning for Mathematical and Code Reasoning},
|
||||||
|
author = {Meena, Naresh},
|
||||||
|
year = {2025},
|
||||||
|
note = {<!-- ADD: arXiv link or report URL -->}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📬 Contact
|
||||||
|
|
||||||
|
<!-- ADD: your contact details / social links -->
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<div align="center">
|
||||||
|
|
||||||
|
*Less data. Less compute. More signal.*
|
||||||
|
|
||||||
|
**[🤗 Model](https://huggingface.co/nareshmeena12/fluxion-370m-instruct) · [💻 Code](<!-- ADD -->) · [📄 Paper](<!-- ADD -->)**
|
||||||
|
|
||||||
|
</div>
|
||||||
15
chat_template.jinja
Normal file
15
chat_template.jinja
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
{% for message in messages %}
|
||||||
|
{% if message['role'] == 'system' %}
|
||||||
|
<|im_start|><|system|>
|
||||||
|
{{ message['content'] }}<|im_end|>
|
||||||
|
{% elif message['role'] == 'user' %}
|
||||||
|
<|im_start|><|user|>
|
||||||
|
{{ message['content'] }}<|im_end|>
|
||||||
|
{% elif message['role'] == 'assistant' %}
|
||||||
|
<|im_start|><|assistant|>
|
||||||
|
{{ message['content'] }}<|im_end|>
|
||||||
|
{% endif %}
|
||||||
|
{% endfor %}
|
||||||
|
{% if add_generation_prompt %}
|
||||||
|
<|im_start|><|assistant|>
|
||||||
|
{% endif %}
|
||||||
17
config.json
Normal file
17
config.json
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"model_type": "llama",
|
||||||
|
"vocab_size": 50000,
|
||||||
|
"hidden_size": 1024,
|
||||||
|
"intermediate_size": 3072,
|
||||||
|
"num_hidden_layers": 26,
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_theta": 500000.0,
|
||||||
|
"max_position_embeddings": 4096,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"torch_dtype": "bfloat16"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:e1380fbe2ec1c2d987d141d14243c77d661f26040ed399e2c6363bcbb444a6e1
|
||||||
|
size 859246896
|
||||||
249126
tokenizer.json
Normal file
249126
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
11
tokenizer_config.json
Normal file
11
tokenizer_config.json
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": "<|bos|>",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|eos|>",
|
||||||
|
"is_local": false,
|
||||||
|
"local_files_only": false,
|
||||||
|
"model_max_length": 4096,
|
||||||
|
"pad_token": "<|pad|>",
|
||||||
|
"tokenizer_class": "PreTrainedTokenizerFast"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user