初始化项目，由ModelHub XC社区提供模型
Model: nareshmeena12/fluxion-370m-instruct Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
364
README.md
Normal file
364
README.md
Normal file
@@ -0,0 +1,364 @@
|
||||
---
|
||||
library_name: transformers
|
||||
tags:
|
||||
- code
|
||||
- math
|
||||
- text-generation-inference
|
||||
- slm
|
||||
- instructmodel
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
---
|
||||
|
||||
# ⚡ Fluxion-370M-Instruct
|
||||
|
||||
### *Trained on 0.227T tokens. Competing with models trained on 28T.*
|
||||
|
||||
<br/>
|
||||
|
||||
[](<!-- ADD -->)
|
||||
[](https://huggingface.co/nareshmeena12/fluxion-370m-instruct)
|
||||
[](<!-- ADD -->)
|
||||
[](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
<div align="center">
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## What is Fluxion?
|
||||
|
||||
**Fluxion-370M-Instruct** is a 370M-parameter instruction-tuned language model built from scratch, focused on **mathematical reasoning and code generation**. Its core design philosophy is simple: *do more with less*.
|
||||
|
||||
Trained on just **227B tokens** — a fraction of what comparable models consume — Fluxion achieves competitive performance on math and coding benchmarks against models that have seen **18–28× more data**. On token efficiency (MATH-500 score per trillion training tokens), Fluxion ranks **1st** across all models in its class by a dominant margin.
|
||||
|
||||
The model went through three training stages: large-scale pretraining, supervised fine-tuning for instruction following, and GRPO reinforcement learning specifically targeting mathematical reasoning with rule-based rewards.
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Token Efficiency — The Core Story
|
||||
|
||||
| Model | Training Tokens | MATH-500 | **MATH-500 / Trillion Tokens** |
|
||||
|:---|:---:|:---:|:---:|
|
||||
| **Fluxion-370M** | **0.227T** | **17.40%** | **🥇 76.7×** |
|
||||
| MobileLLM-R1-360M | 4.2T | 25.60% | 6.1× |
|
||||
| Gemma3-270M | 6T | 8.20% | 1.4× |
|
||||
| Granite-350M | 11T | 19.20% | 1.7× |
|
||||
| Qwen2.5-0.5B | 18T | 29.80% | 1.7× |
|
||||
| LFM2.5-350M | 28T | 13.40% | 0.5× |
|
||||
| SmolLM2-360M | 4T | 3.40% | 0.9× |
|
||||
|
||||
**Fluxion delivers 76.7× higher MATH-500 score per trillion training tokens than any other model in this comparison.** This is a structural advantage from architecture choices, custom tokenization, and a high-signal data pipeline — not a measurement artifact.
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
Fluxion is a clean, modern decoder-only transformer. No hybrid layers, no custom operators — just well-chosen standard components assembled deliberately.
|
||||
|
||||
```
|
||||
FluxionForCausalLM
|
||||
├── Embedding (vocab_size=50,000 · hidden_size=1024)
|
||||
├── 26 × DecoderBlock
|
||||
│ ├── RMSNorm (ε = 1e-6)
|
||||
│ ├── GroupedQueryAttention
|
||||
│ │ ├── 16 Query heads · 8 KV heads (GQA 2:1)
|
||||
│ │ └── RoPE (θ = 500,000) + NTK scaling
|
||||
│ ├── RMSNorm (ε = 1e-6)
|
||||
│ └── SwiGLU FFN (hidden → 3072 → hidden)
|
||||
└── RMSNorm → LM Head (weight-tied to embedding)
|
||||
```
|
||||
|
||||
### Hyperparameters
|
||||
|
||||
| Parameter | Value |
|
||||
|:---|:---:|
|
||||
| **Total Parameters** | ~370M |
|
||||
| **Layers** | 26 |
|
||||
| **Hidden Size** | 1024 |
|
||||
| **FFN Intermediate Size** | 3072 |
|
||||
| **Attention Heads (Q)** | 16 |
|
||||
| **KV Heads (GQA)** | 8 |
|
||||
| **GQA Group Ratio** | 2 : 1 |
|
||||
| **Head Dimension** | 64 |
|
||||
| **Context Length** | 4096 |
|
||||
| **Vocabulary Size** | 50,000 |
|
||||
| **RoPE Base θ** | 500,000 |
|
||||
| **RoPE Scaling** | NTK |
|
||||
| **Normalization** | RMSNorm (ε = 1e-6) |
|
||||
| **Activation** | SwiGLU |
|
||||
| **Attention** | Flash Attention |
|
||||
| **Weight Tying** | Yes (embed ↔ lm\_head) |
|
||||
| **Attention Bias** | No |
|
||||
| **FFN Bias** | No |
|
||||
| **dtype** | bfloat16 |
|
||||
|
||||
### Design Choices & Rationale
|
||||
|
||||
**Grouped Query Attention (16Q / 8KV):** Halves KV cache memory vs standard MHA with no meaningful quality loss at this scale. Larger effective batch sizes at both training and inference time.
|
||||
|
||||
**RoPE with θ = 500,000 + NTK scaling:** High base frequency improves length generalisation beyond the training window. NTK scaling further stabilises attention patterns at longer sequences without fine-tuning.
|
||||
|
||||
**SwiGLU activation:** Consistently stronger than GeLU/ReLU on reasoning tasks. The gated structure gives the FFN richer expressiveness per parameter.
|
||||
|
||||
**Custom BPE tokenizer (50k vocab):** Trained from scratch on 100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. A smaller, domain-tuned vocabulary means fewer tokens per math expression and code snippet, directly improving effective context utilisation and reducing sequence lengths vs generic 128k tokenizers.
|
||||
|
||||
**Weight tying (embed ↔ lm\_head):** Saves ~50M parameters, keeps the model compact, and forces the input and output representation spaces to align.
|
||||
|
||||
**Flash Attention:** Full Flash Attention used throughout training for memory efficiency and throughput. TF32 enabled for matrix multiplications.
|
||||
|
||||
---
|
||||
|
||||
## 📊 Benchmarks
|
||||
|
||||
All results are **0-shot** unless noted. Evaluation is strict.
|
||||
|
||||
### Math & Code
|
||||
|
||||
| Model | Tokens | GSM8K | MATH-500 | HumanEval | MBPP | **Avg** |
|
||||
|:---|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| Qwen2.5-0.5B | 18T | 43.44% | 29.80% | 30.49% | 38.91% | 35.66% |
|
||||
| Granite-350M | 11T | 31.69% | 19.20% | 29.88% | 42.02% | 30.70% |
|
||||
| LFM2.5-350M | 28T | 32.68% | 13.40% | 10.37% | 5.45% | 15.48% |
|
||||
| MobileLLM-R1-360M | 4.2T | 23.81% | 25.60% | 19.51% | 17.51% | 21.61% |
|
||||
| **Fluxion-370M** | **0.227T** | **29.57%** | **17.40%** | **16.46%** | **20.23%** | **20.92%** |
|
||||
| SmolLM2-360M | 4T | 8.87% | 3.40% | 17.68% | 35.02% | 16.24% |
|
||||
| Gemma3-270M | 6T | 7.58% | 8.20% | 11.59% | 19.07% | 11.61% |
|
||||
|
||||
### General Reasoning
|
||||
|
||||
| Model | ARC | PIQA | OBQA | CSQA | MMLU | Wino | IFEval | **Avg** |
|
||||
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| Qwen2.5-0.5B | 50.9% | 70.1% | 36.2% | 42.3% | 45.9% | 51.1% | 74.1% | 52.9% |
|
||||
| Granite-350M | 51.8% | 68.9% | 37.8% | 40.5% | 33.8% | 53.8% | 83.2% | 52.8% |
|
||||
| LFM2.5-350M | 49.4% | 67.0% | 31.6% | 33.4% | 42.2% | 52.0% | 88.3% | 52.0% |
|
||||
| SmolLM2-360M | 45.9% | 71.5% | 35.2% | 29.0% | 24.9% | 54.8% | 77.5% | 48.4% |
|
||||
| Gemma3-270M | 43.7% | 66.6% | 30.6% | 36.9% | 24.7% | 49.7% | 71.5% | 46.2% |
|
||||
| MobileLLM-R1-360M | 39.1% | 58.6% | 26.2% | 27.6% | 26.7% | 50.8% | 71.2% | 42.9% |
|
||||
| **Fluxion-370M** | **36.3%** | **61.6%** | **29.4%** | **27.3%** | **25.3%** | **51.9%** | **66.9%** | **42.7%** |
|
||||
|
||||
### Rankings Summary
|
||||
|
||||
| Category | Score | Rank |
|
||||
|:---|:---:|:---:|
|
||||
| Math Average (GSM8K + MATH-500) | 23.49% | **4th** |
|
||||
| Code Average (HumanEval + MBPP) | 18.35% | **5th** |
|
||||
| Math + Code Combined | 20.92% | **4th** |
|
||||
| General Average | 39.19% | 7th |
|
||||
| **Token Efficiency (MATH-500 / T tokens)** | **76.7×** | **🥇 1st** |
|
||||
|
||||
> Fluxion's design target is math and code under tight data constraints. On general benchmarks like IFEval and MMLU it trails models trained on significantly more diverse data — this is expected and intentional. On token efficiency it has no close competitor.
|
||||
|
||||
---
|
||||
|
||||
## 💬 Chat Template & Tokenizer
|
||||
|
||||
Fluxion-Instruct uses the **ChatML** format:
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
You are a helpful assistant.<|im_end|>
|
||||
<|im_start|>user
|
||||
{your prompt here}<|im_end|>
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
### Special Tokens
|
||||
|
||||
| Token | ID | Role |
|
||||
|:---|:---:|:---|
|
||||
| `<\|im_start\|>` | 100278 | Turn start marker |
|
||||
| `<\|im_end\|>` | 100279 | Turn end marker |
|
||||
| `<\|pad\|>` | — | Padding token |
|
||||
|
||||
### Stopping Token IDs
|
||||
|
||||
Always pass `eos_token_id=[1, 4]` during generation. Without these, the model may continue generating beyond the intended response boundary:
|
||||
|
||||
```python
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
eos_token_id=[1, 4],
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
Token ID `1` is the primary EOS token and token ID `4` is the `<|im_end|>` marker. Including both ensures clean turn termination in all inference scenarios.
|
||||
|
||||
### Tokenizer
|
||||
|
||||
Fluxion uses a **custom BPE tokenizer** trained from scratch on ~100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. With a **50,000 token vocabulary**, it is significantly more compact than general-purpose tokenizers (32k–128k). This means:
|
||||
|
||||
- Fewer tokens per math expression and code snippet
|
||||
- Better effective context utilisation within the 4096 token window
|
||||
- Faster embedding lookups and a smaller lm\_head projection
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
"nareshmeena12/fluxion-370m-instruct",
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(
|
||||
"nareshmeena12/fluxion-370m-instruct"
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are a helpful assistant that solves math problems step by step."},
|
||||
{"role": "user", "content": "What is the sum of the first 100 natural numbers?"}
|
||||
]
|
||||
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True
|
||||
)
|
||||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=512,
|
||||
eos_token_id=[1, 4],
|
||||
do_sample=True,
|
||||
top_p=0.9,
|
||||
)
|
||||
|
||||
response = tokenizer.decode(
|
||||
output[0, inputs["input_ids"].shape[1]:],
|
||||
skip_special_tokens=True
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
|
||||
|
||||
> **Note:** Always include `eos_token_id=[1, 4]` in all `generate()` calls regardless of quantization mode.
|
||||
|
||||
---
|
||||
|
||||
## 🗂️ Training
|
||||
|
||||
Fluxion was trained in three stages.
|
||||
|
||||
### Stage 1 — Pretraining
|
||||
|
||||
Large-scale unsupervised pretraining on a curated mix of math, code, and English text.
|
||||
|
||||
| Parameter | Value |
|
||||
|:---|:---:|
|
||||
| **Total Tokens** | ~227B (0.227T) |
|
||||
| **Steps** | 264,000 |
|
||||
| **Batch Size** | 10 sequences |
|
||||
| **Gradient Accumulation** | 21 steps |
|
||||
| **Effective Batch** | ~860K tokens/step |
|
||||
| **Learning Rate** | 3e-4 |
|
||||
| **Min Learning Rate** | 9e-5 |
|
||||
| **LR Schedule** | Cosine decay |
|
||||
| **Warmup Steps** | 2,000 |
|
||||
| **Sequence Length** | 4,096 |
|
||||
| **Optimizer** | AdamW |
|
||||
| **Precision** | bfloat16 + TF32 |
|
||||
| **Attention** | Flash Attention |
|
||||
| **Positional Encoding** | RoPE (θ=500k) + NTK scaling |
|
||||
|
||||
**Pretraining data:** *(to be added)*
|
||||
|
||||
### Stage 2 — Supervised Fine-Tuning (SFT)
|
||||
|
||||
Instruction tuning using ChatML format to teach the model to follow user instructions across math, code, and general tasks.
|
||||
|
||||
| Parameter | Value |
|
||||
|:---|:---:|
|
||||
| **Format** | ChatML |
|
||||
| **Data** | *(to be added)* |
|
||||
|
||||
### Stage 3 — GRPO (Math Reasoning)
|
||||
|
||||
Group Relative Policy Optimisation applied specifically to strengthen mathematical reasoning.
|
||||
|
||||
| Parameter | Value |
|
||||
|:---|:---:|
|
||||
| **Algorithm** | GRPO |
|
||||
| **Reward Signal** | Rule-based answer correctness |
|
||||
| **Target Domain** | Mathematics |
|
||||
| **Data** | *(to be added)* |
|
||||
|
||||
Rule-based rewards verify final answer correctness against ground truth — no LLM judge, no model-in-the-loop. This keeps the reward signal clean, fast, and free from reward hacking through verbosity or style.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Intended Use
|
||||
|
||||
**Fluxion-370M-Instruct is designed for:**
|
||||
- Mathematical problem solving and step-by-step reasoning
|
||||
- Code generation and explanation (Python, general purpose)
|
||||
- Instruction-following tasks in math and code domains
|
||||
- Fine-tuning base for domain-specific math/code applications
|
||||
- Efficient deployment — capable performance on modest hardware
|
||||
- Research into token-efficient pretraining and RL for reasoning
|
||||
|
||||
**Fluxion-370M-Instruct is NOT designed for:**
|
||||
- Knowledge-intensive QA requiring broad world knowledge
|
||||
- Long-document tasks beyond 4096 tokens
|
||||
- Multilingual use cases
|
||||
- Safety-critical production deployments without additional alignment
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Limitations
|
||||
|
||||
- **Narrow training data scope.** 0.227T tokens is intentionally small and domain-focused. The model has limited world knowledge outside math, code, and general English.
|
||||
- **Context length is 4096 tokens.** Not suitable for long-document tasks without chunking strategies.
|
||||
- **General benchmarks lag.** IFEval (66.9%) and MMLU (25.3%) reflect that the model was not optimised for instruction following diversity or broad knowledge recall — this is a deliberate trade-off.
|
||||
- **Custom tokenizer.** The 50k BPE vocabulary is domain-tuned. Rare tokens, non-English text, and highly specialised symbols outside the training distribution may tokenise suboptimally.
|
||||
- **Small scale.** At 370M parameters, the model is outperformed by larger models on complex multi-step reasoning chains requiring deep world knowledge.
|
||||
- **Always set `eos_token_id=[1, 4]`.** Without explicit stopping token IDs the model may continue generating past the intended response boundary.
|
||||
|
||||
---
|
||||
|
||||
## 📰 News
|
||||
|
||||
- `<!-- ADD DATE -->` — Fluxion-370M-Instruct released on Hugging Face
|
||||
- `<!-- ADD DATE -->` — Training code released on GitHub
|
||||
- `<!-- ADD DATE -->` — Technical report available on arXiv
|
||||
|
||||
---
|
||||
|
||||
## 📄 Citation
|
||||
|
||||
```bibtex
|
||||
@misc{fluxion2025,
|
||||
title = {Fluxion-370M: Token-Efficient Pretraining and Reinforcement Learning for Mathematical and Code Reasoning},
|
||||
author = {Meena, Naresh},
|
||||
year = {2025},
|
||||
note = {<!-- ADD: arXiv link or report URL -->}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📬 Contact
|
||||
|
||||
<!-- ADD: your contact details / social links -->
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
|
||||
*Less data. Less compute. More signal.*
|
||||
|
||||
**[🤗 Model](https://huggingface.co/nareshmeena12/fluxion-370m-instruct) · [💻 Code](<!-- ADD -->) · [📄 Paper](<!-- ADD -->)**
|
||||
|
||||
</div>
|
||||
15
chat_template.jinja
Normal file
15
chat_template.jinja
Normal file
@@ -0,0 +1,15 @@
|
||||
{% for message in messages %}
|
||||
{% if message['role'] == 'system' %}
|
||||
<|im_start|><|system|>
|
||||
{{ message['content'] }}<|im_end|>
|
||||
{% elif message['role'] == 'user' %}
|
||||
<|im_start|><|user|>
|
||||
{{ message['content'] }}<|im_end|>
|
||||
{% elif message['role'] == 'assistant' %}
|
||||
<|im_start|><|assistant|>
|
||||
{{ message['content'] }}<|im_end|>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% if add_generation_prompt %}
|
||||
<|im_start|><|assistant|>
|
||||
{% endif %}
|
||||
17
config.json
Normal file
17
config.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"model_type": "llama",
|
||||
"vocab_size": 50000,
|
||||
"hidden_size": 1024,
|
||||
"intermediate_size": 3072,
|
||||
"num_hidden_layers": 26,
|
||||
"num_attention_heads": 16,
|
||||
"num_key_value_heads": 8,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_theta": 500000.0,
|
||||
"max_position_embeddings": 4096,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e1380fbe2ec1c2d987d141d14243c77d661f26040ed399e2c6363bcbb444a6e1
|
||||
size 859246896
|
||||
249126
tokenizer.json
Normal file
249126
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
11
tokenizer_config.json
Normal file
11
tokenizer_config.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<|bos|>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|eos|>",
|
||||
"is_local": false,
|
||||
"local_files_only": false,
|
||||
"model_max_length": 4096,
|
||||
"pad_token": "<|pad|>",
|
||||
"tokenizer_class": "PreTrainedTokenizerFast"
|
||||
}
|
||||
Reference in New Issue
Block a user