147 lines
4.9 KiB
Markdown
147 lines
4.9 KiB
Markdown
---
|
||
license: apache-2.0
|
||
datasets:
|
||
- HuggingFaceFW/fineweb-edu
|
||
- mlfoundations/dclm-baseline-1.0
|
||
- HuggingFaceTB/finemath
|
||
- HuggingFaceCode/stack-v3-train
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
base_model: HuggingFaceTB/SmolLM2-135M
|
||
tags:
|
||
- causal-lm
|
||
- language-model
|
||
- base-model
|
||
- llama
|
||
- smollm
|
||
- pytorch
|
||
---
|
||
|
||
# Kiyo-135M
|
||
|
||
Kiyo-135M is a decoder-only language model pretrained from scratch on 200B tokens drawn from [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0), [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath), and [Stack-v3-train](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train).
|
||
|
||
The model follows the [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) architecture: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings. SmolLM2 was chosen as a base architecture because it is specifically tuned for small-scale pretraining efficiency — narrow hidden size with a comparatively deep layer stack, and a large 49k-token vocabulary that keeps sequence lengths short for a model this size. Kiyo-135M reuses this architecture but is trained independently from a random initialization on its own data mixture, rather than starting from SmolLM2's own weights.
|
||
|
||
## Model Details
|
||
|
||
| Field | Value |
|
||
|---|---:|
|
||
| Parameters | 134,515,008 |
|
||
| Architecture | Llama-style decoder (SmolLM2 architecture) |
|
||
| Layers | 30 |
|
||
| Hidden size | 576 |
|
||
| Intermediate size | 1,536 |
|
||
| Attention heads | 9 |
|
||
| KV heads | 3 |
|
||
| Attention type | Grouped query attention |
|
||
| Activation | SwiGLU |
|
||
| Normalization | RMSNorm |
|
||
| Positional encoding | RoPE (theta 100,000) |
|
||
| Vocabulary size | 49,152 |
|
||
| Context length | 8,192 |
|
||
| Embeddings | Tied input/output embeddings |
|
||
| Training tokens | 200,000,000,000 |
|
||
| Weight format | safetensors |
|
||
|
||
## Training Data
|
||
|
||
| Source | Domain |
|
||
|---|---|
|
||
| FineWeb-Edu | General web text, education-filtered |
|
||
| DCLM-Baseline | General web text, high-quality filtered |
|
||
| FineMath | Mathematical reasoning |
|
||
| Stack-v3-train | Source code |
|
||
|
||
## Benchmarks
|
||
|
||
Self-reported results from the official [BananaMind Base Bench 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1) script, all measured with the same runner, dtype (bfloat16) and GPU.
|
||
|
||
| Model | Params | Overall Elo |
|
||
|---|---:|---:|
|
||
| **Kiyo-135M** | **134.5M** | **1,126** |
|
||
| BananaMind-2-Pro | 139.0M | 1,124 |
|
||
| Rose-Pro | 151.3M | 1,105 |
|
||
| GPT-2 | 124M | 990 |
|
||
|
||
Figures for BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite.
|
||
|
||
**Detailed Kiyo-135M result**
|
||
|
||
| Category | Accuracy | z vs. chance | Elo | Significant |
|
||
|---|---:|---:|---:|:---:|
|
||
| Language completion | 100.0% | +12.25 | 1,570 | * |
|
||
| Code completion | 86.0% | +9.96 | 1,420 | * |
|
||
| World knowledge | 80.0% | +8.98 | 1,151 | * |
|
||
| Commonsense | 74.0% | +8.00 | 1,110 | * |
|
||
| Logical reasoning | 58.0% | +5.39 | 1,118 | * |
|
||
| Context tracking | 44.0% | +3.10 | 914 | * |
|
||
| Quantitative | 32.0% | +1.14 | 913 | |
|
||
|
||
\* = passes 1.96σ vs. chance; n=50 per category
|
||
|
||
**By difficulty**
|
||
|
||
| Difficulty | Accuracy |
|
||
|---|---:|
|
||
| Easy | 76.9% |
|
||
| Medium | 69.2% |
|
||
| Hard | 56.9% |
|
||
|
||
**Summary**
|
||
|
||
| Metric | Value |
|
||
|---|---:|
|
||
| Parameters | 134,515,008 |
|
||
| Overall Elo | 1,126 |
|
||
| Chance floor | 805 |
|
||
| Above chance floor | +321 |
|
||
| Raw accuracy | 67.7% |
|
||
|
||
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
pip install -U transformers safetensors torch
|
||
```
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
||
model_id = "DedeProGames/Kiyo-135M"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
|
||
).cuda().eval()
|
||
|
||
prompt = "The meaning of life is "
|
||
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
|
||
|
||
with torch.no_grad():
|
||
output = model.generate(
|
||
input_ids=input_ids,
|
||
max_new_tokens=64,
|
||
do_sample=False,
|
||
repetition_penalty=1.1,
|
||
pad_token_id=tokenizer.eos_token_id,
|
||
eos_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Limitations
|
||
|
||
This is a base model, not instruction-tuned — it continues text rather than following instructions. At 135M parameters it produces fluent, well-structured text and is strong on language completion and code, but accuracy drops on quantitative and multi-step context-tracking tasks. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
|
||
|
||
## License
|
||
|
||
Apache 2.0
|