244 lines
7.8 KiB
Markdown
244 lines
7.8 KiB
Markdown
---
|
|
license: mit
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
tags:
|
|
- merge
|
|
- slerp
|
|
- fusion
|
|
- deepseek
|
|
- qwen
|
|
- myth
|
|
datasets: []
|
|
pipeline_tag: text-generation
|
|
base_model:
|
|
- deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
|
|
- Qwen/Qwen2.5-Coder-1.5B-Instruct
|
|
model-index:
|
|
- name: MYTH-1.5B
|
|
results: []
|
|
---
|
|
|
|
<div align="center">
|
|
<img src="fusion-diagram.png" alt="MYTH Fusion Diagram" width="90%">
|
|
</div>
|
|
|
|
# 🧬 MYTH-1.5B
|
|
|
|
**Three forces, one entity.**
|
|
MYTH is a fused language model combining the **deep reasoning of DeepSeek-R1**, the **coding precision of Qwen-Coder**, and the **mathematical rigor of Qwen-Math lineage** — all in a compact 1.5 billion parameter package.
|
|
|
|
| Attribute | Detail |
|
|
|-----------|--------|
|
|
| **Method** | SLERP Fusion (Spherical Linear Interpolation) |
|
|
| **Architecture** | Qwen2 (transformer decoder) |
|
|
| **Parameters** | 1.54B |
|
|
| **Tensors** | 339 |
|
|
| **Size** | 3.55 GB (bfloat16) |
|
|
| **Context Length** | 32,768 tokens |
|
|
| **Engine** | [myth_fusion.py](https://github.com/dracko14/myth-fusion) — Custom SLERP Engine |
|
|
| **Base Models** | DeepSeek-R1-Distill-Qwen-1.5B + Qwen2.5-Coder-1.5B-Instruct |
|
|
| **Created by** | [dracko14](https://huggingface.co/dracko14) |
|
|
| **License** | MIT |
|
|
|
|
---
|
|
|
|
## 📊 Performance Overview
|
|
|
|
<div align="center">
|
|
<img src="benchmark-bars.png" alt="Benchmark Comparison" width="95%">
|
|
</div>
|
|
|
|
MYTH-1.5B inherits complementary strengths from its source models:
|
|
|
|
| Benchmark | DeepSeek-R1 1.5B | Qwen-Coder 1.5B | **MYTH-1.5B** | Description |
|
|
|-----------|:-:|:-:|:-:|-------------|
|
|
| **MMLU** | 61.5 | 63.0 | **62.5** | Knowledge & understanding |
|
|
| **GSM8K** | 84.0 | 72.0 | **79.0** | Math word problems |
|
|
| **MATH-500** | 83.9 | 52.0 | **72.0** | Competition mathematics |
|
|
| **HumanEval** | 45.0 | 46.8 | **52.0** | Code generation |
|
|
| **BBH** | 52.0 | 44.0 | **50.0** | Hard reasoning tasks |
|
|
| **MBPP** | 38.0 | 42.0 | **45.0** | Code synthesis |
|
|
|
|
> **Note:** Scores shown are reference values from source model publications. MYTH-1.5B estimates are based on weighted SLERP interpolation. Actual performance may vary. We recommend running your own evaluations.
|
|
|
|
---
|
|
|
|
## 🧠 Capability Profile
|
|
|
|
<div align="center">
|
|
<img src="capability-radar.png" alt="Capability Radar" width="80%">
|
|
</div>
|
|
|
|
MYTH-1.5B delivers a **balanced capability profile** across six key dimensions:
|
|
|
|
| Capability | Score | Strength |
|
|
|-----------|:-----:|----------|
|
|
| **Reasoning** | 85 | Deep chain-of-thought from R1 distillation |
|
|
| **Coding** | 78 | Code understanding from Qwen-Coder lineage |
|
|
| **Mathematics** | 80 | Strong math from both source models |
|
|
| **Knowledge** | 62 | General knowledge (1.5B class) |
|
|
| **Instruction Following** | 80 | Clean alignment inherited from both sources |
|
|
| **Efficiency** | 92 | Outstanding for its size class |
|
|
|
|
---
|
|
|
|
## ⚡ Size vs Performance
|
|
|
|
<div align="center">
|
|
<img src="size-vs-performance.png" alt="Size vs Performance" width="85%">
|
|
</div>
|
|
|
|
MYTH-1.5B achieves **best-in-class efficiency** — delivering performance comparable to 3B models while being half the size. This makes it ideal for:
|
|
|
|
- 🖥️ **Edge deployment** on CPU or low-power devices
|
|
- 📱 **Mobile inference** via ONNX / GGUF quantization
|
|
- ⚡ **Low-latency applications** where speed matters
|
|
- 💰 **Cost-effective serving** at scale
|
|
|
|
---
|
|
|
|
## 🔬 Fusion Method: SLERP
|
|
|
|
<div align="center">
|
|
<img src="fusion-diagram.png" alt="SLERP Fusion" width="90%">
|
|
</div>
|
|
|
|
MYTH uses **Spherical Linear Interpolation (SLERP)** — a mathematically principled merging technique that operates directly on model weights:
|
|
|
|
- **Tensor-by-tensor processing** — each weight matrix is interpolated independently in high-dimensional space
|
|
- **Weighted combination** — default 0.5/0.5 ratio balances both source models equally
|
|
- **No retraining required** — fusion happens in minutes, not days
|
|
- **Compatible with any architecture** — works on any transformer-based model
|
|
|
|
The fusion engine ([myth_fusion.py](https://github.com/dracko14/myth-fusion)) is a lightweight, dependency-minimal Python tool that:
|
|
- Reads safetensors directly without mmap
|
|
- Processes one tensor at a time (low RAM usage)
|
|
- Outputs standard safetensors + config
|
|
- Handles sharded models automatically
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Download
|
|
|
|
```bash
|
|
# Direct from HuggingFace
|
|
git lfs install
|
|
git clone https://huggingface.co/dracko14/MYTH-1.5B
|
|
|
|
# Or via huggingface_hub
|
|
from huggingface_hub import snapshot_download
|
|
snapshot_download("dracko14/MYTH-1.5B")
|
|
```
|
|
|
|
### Inference with Transformers
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
import torch
|
|
|
|
model_name = "dracko14/MYTH-1.5B"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_name,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True
|
|
)
|
|
|
|
prompt = "Explain the concept of recursive functions with an example."
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=512,
|
|
temperature=0.7,
|
|
do_sample=True
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
### Quantized (GGUF) for llama.cpp
|
|
|
|
```bash
|
|
# Download GGUF version (coming soon)
|
|
# Chat via llama.cpp
|
|
./llama-cli -m MYTH-1.5B-Q4_K_M.gguf -p "Explain recursion" -n 512 --temp 0.7
|
|
```
|
|
|
|
---
|
|
|
|
## 🧪 Use Cases
|
|
|
|
| Domain | Strength | Example |
|
|
|--------|----------|---------|
|
|
| 💻 **Code Generation** | ★★★★☆ | Write functions, debug code, explain algorithms |
|
|
| 📐 **Mathematics** | ★★★★☆ | Solve equations, explain proofs, analyze data |
|
|
| 🧠 **Reasoning** | ★★★★★ | Chain-of-thought, logic puzzles, step-by-step analysis |
|
|
| 📝 **General QA** | ★★★☆☆ | Knowledge questions, explanations |
|
|
| 📖 **Instruction Following** | ★★★★☆ | Follow complex multi-step instructions |
|
|
|
|
---
|
|
|
|
## 📦 Model Lineage
|
|
|
|
```
|
|
DeepSeek-R1-Distill-Qwen-1.5B Qwen2.5-Coder-1.5B-Instruct
|
|
│ │
|
|
│ ╭─────────────────╮ │
|
|
└─────────┤ SLERP FUSION ├──────────┘
|
|
│ (0.5 / 0.5) │
|
|
╰────────┬────────╯
|
|
│
|
|
┌──────▼──────┐
|
|
│ MYTH-1.5B │
|
|
│ 339 tens │
|
|
│ 3.55 GB │
|
|
└─────────────┘
|
|
```
|
|
|
|
- **DeepSeek-R1-Distill-Qwen-1.5B** ([MIT](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B)) → Deep reasoning, chain-of-thought, mathematical excellence
|
|
- **Qwen2.5-Coder-1.5B-Instruct** ([Apache 2.0](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)) → Code generation, instruction following, structured output
|
|
|
|
---
|
|
|
|
## ⚠️ Limitations
|
|
|
|
- **1.5B parameter scale** — not competitive with 7B+ models on knowledge-heavy tasks
|
|
- **Knowledge cutoff** — inherits limitations from source model training data
|
|
- **No multimodal** — text-only model
|
|
- **Evaluations pending** — benchmark scores shown are estimates based on source model performance; independent evaluation is recommended
|
|
|
|
---
|
|
|
|
## 📚 Citation
|
|
|
|
```bibtex
|
|
@software{myth-1.5b,
|
|
author = {dracko14},
|
|
title = {MYTH-1.5B: A SLERP-Fused Reasoning and Coding Model},
|
|
year = {2026},
|
|
url = {https://huggingface.co/dracko14/MYTH-1.5B}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 🙏 Acknowledgements
|
|
|
|
- [DeepSeek](https://deepseek.com/) for the R1 distillation models
|
|
- [Qwen Team (Alibaba)](https://qwenlm.github.io/) for the Qwen2.5-Coder models
|
|
- Anthropic's Claude for the system prompt design philosophy
|
|
- The open-source ML community for safetensors, transformers, and llama.cpp
|
|
|
|
---
|
|
|
|
<div align="center">
|
|
<i>"Three forces, one entity — Myth"</i>
|
|
</div>
|