57 lines
2.1 KiB
Markdown
57 lines
2.1 KiB
Markdown
|
---
|
||
|
|
license: mit
|
||
|
|
datasets:
|
||
|
|
- HuggingFaceTB/smollm-corpus
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- moe
|
||
|
|
- causal-lm
|
||
|
|
- mistral
|
||
|
|
- tiny-moe
|
||
|
|
---
|
||
|
|
|
||
|
|
# TinyMoE-200M-2x16
|
||
|
|
|
||
|
|
**TinyMoE-200M-2x16** is a compact, highly efficient Sparse Mixture of Experts (MoE) language model built upon the Mixtral/Mistral architecture. Designed for research, edge applications, and resource-constrained environments, this model leverages an expert-routing mechanism to balance a larger total parameter capacity with ultra-low computational overhead during inference.
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
* **Architecture:** Sparse Mixture of Experts (MoE)
|
||
|
|
* **Total Parameters:** ~200m total parameters
|
||
|
|
* **Active Parameters per Token:** ~50M active parameters
|
||
|
|
* **Expert Configuration:** 16 total local experts, 2 active experts routed per token (`num_experts_per_tok": 2`)
|
||
|
|
* **Context Length:** 1024 tokens
|
||
|
|
* **Base Architecture:** Mixtral / Mistral For Causal LM
|
||
|
|
* **License:** MIT
|
||
|
|
|
||
|
|
## Training Data
|
||
|
|
|
||
|
|
This model was trained on`FineWeb-Edu-Dedup` 60% and `Cosmopedia-v2` 40%.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Quick Start
|
||
|
|
|
||
|
|
You can load and experiment with this model using the Hugging Face `transformers` library:
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model_id = "FlameF0X/TinyMoE-200m-2x16"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
|
|
|
||
|
|
input_text = "Wikipedia is a free"
|
||
|
|
inputs = tokenizer(input_text, return_tensors="pt")
|
||
|
|
|
||
|
|
outputs = model.generate(**inputs, max_new_tokens=50)
|
||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
|
|
```
|