70 lines
1.8 KiB
Markdown
70 lines
1.8 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
license: apache-2.0
|
|||
|
|
tags:
|
|||
|
|
- causal-lm
|
|||
|
|
- llama
|
|||
|
|
- sable
|
|||
|
|
- small-language-model
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 🖤 Sable-Mini-30M (Preview)
|
|||
|
|
|
|||
|
|
**Sable-Mini** is a 30M parameter causal language model trained on approximately **3–4B tokens**. It is the flagship of the Sable tiny model family, balancing compact size with capable text generation.
|
|||
|
|
|
|||
|
|
> ⚠️ **This is a preview release** — expect improvements in future versions.
|
|||
|
|
|
|||
|
|
## 📊 Benchmark Results
|
|||
|
|
|
|||
|
|
Evaluated with the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness):
|
|||
|
|
|
|||
|
|
| Benchmark | Accuracy | Metric |
|
|||
|
|
| :--- | :---: | :---: |
|
|||
|
|
| **BoolQ** | **57.5%** | `acc` |
|
|||
|
|
| **PIQA** | **57.8%** | `acc_norm` |
|
|||
|
|
| **WinoGrande** | **53.3%** | `acc` |
|
|||
|
|
| **ARC-Easy** | **39.0%** | `acc_norm` |
|
|||
|
|
| **HellaSwag** | **28.1%** | `acc_norm` |
|
|||
|
|
| **Lambada** | **11.7%** | `acc` |
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## 🚀 Quick Start
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained("yourorg/sable-mini-30m-preview")
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("yourorg/sable-mini-30m-preview")
|
|||
|
|
|
|||
|
|
inputs = tokenizer("The future of tiny models is", return_tensors="pt")
|
|||
|
|
outputs = model.generate(**inputs, max_new_tokens=50)
|
|||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 🏗️ Model Details
|
|||
|
|
|
|||
|
|
| Property | Value |
|
|||
|
|
|----------|-------|
|
|||
|
|
| Parameters | 30M |
|
|||
|
|
| Architecture | Llama-style decoder |
|
|||
|
|
| Training tokens | ~3–4B |
|
|||
|
|
| Context length | ← add yours |
|
|||
|
|
| Vocab size | ← add yours |
|
|||
|
|
|
|||
|
|
## ⚠️ Limitations
|
|||
|
|
|
|||
|
|
- Preview release — not recommended for production use
|
|||
|
|
- Limited knowledge due to small parameter count
|
|||
|
|
- May produce biased or incorrect outputs
|
|||
|
|
|
|||
|
|
## 📜 License
|
|||
|
|
|
|||
|
|
Apache 2.0 — free for commercial and research use.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
*Part of the Sable family 🖤 — tiny models, sharp minds.*
|