124 lines
4.4 KiB
Markdown
124 lines
4.4 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
- code
|
||
license: apache-2.0
|
||
tags:
|
||
- smol
|
||
- pretraining
|
||
- instruct
|
||
- 50M
|
||
- causal-lm
|
||
- gqa
|
||
- swiglu
|
||
- rmsnorm
|
||
datasets:
|
||
- HuggingFaceTB/smollm-corpus
|
||
metrics:
|
||
- perplexity
|
||
model-index:
|
||
- name: Quark-50m-Instruct
|
||
results: []
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Quark-50m-Instruct
|
||
|
||
**Quark-50m-Instruct** is a small (≈56M parameters) decoder-only language model, fine-tuned for instruction following.
|
||
It is built on the same architecture of “SmolLM” family and was fully pretrained on 5 billion tokens from
|
||
[HuggingFaceTB/smollm‑corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus).
|
||
|
||
- **Model type:** Causal Language Model (LLaMA‑style decoder)
|
||
- **Architecture:** GQA · SwiGLU · RMSNorm · RoPE · Weight‑tying
|
||
- **Pretraining tokens:** 5 B
|
||
- **Fine‑tuning:** Instruction‑tuned (details below)
|
||
- **Creators:** [OvercastLab](https://huggingface.co/OvercastLab) (research & development lab for ML/AI)
|
||
- **Release date:** 22 April 2026
|
||
|
||
## Model Summary
|
||
|
||
Quark-50m-Instruct is designed to be an efficient assistant that can run on consumer GPUs (e.g., RTX 3070 with 8 GB VRAM)
|
||
and even on CPU for light workloads. It is **not** competitive with large models on knowledge‑intensive tasks,
|
||
but it excels at:
|
||
|
||
- Simple conversational tasks
|
||
- Code generation and explanation (Python)
|
||
- Short text rewriting and summarisation
|
||
- On‑device / edge inference
|
||
|
||
The architecture closely follows the efficient‑small‑LM blueprint popularised by SmolLM:
|
||
|
||
| Component | Details |
|
||
|-------------|-------------------------------|
|
||
| Vocab size | 49,152 |
|
||
| Hidden size | 384 |
|
||
| Layers | 24 |
|
||
| Attention | Grouped Query (6 Q heads, 2 KV heads) |
|
||
| FFN | SwiGLU with 1,024 intermediate |
|
||
| Position | RoPE (θ = 10,000) |
|
||
| Normalisation | RMSNorm (pre‑block) |
|
||
|
||
Total trainable parameters: **≈48 M** (with weight tying).
|
||
|
||
### Benchmark Evaluation Metrics
|
||
|
||
| Category | Benchmark | Metric | Score / Value | Status |
|
||
| :--- | :--- | :--- | :---: | :---: |
|
||
| **Linguistics & Grammar** | BLiMP | Accuracy | 68.12% | Success |
|
||
| **Commonsense & Reasoning** | PIQA | Normalized Accuracy | 57.83% | Success |
|
||
| | COPA | Accuracy | 57.00% | Success |
|
||
| | BoolQ | Accuracy | 52.17% | Success |
|
||
| | WinoGrande | Accuracy | 47.36% | Success |
|
||
| | HellaSwag | Normalized Accuracy | 28.49% | Success |
|
||
| | RACE | Accuracy | 26.41% | Success |
|
||
| | CommonsenseQA | Accuracy | 20.31% | Success |
|
||
| **Academic & Knowledge** | SciQ | Normalized Accuracy | 49.00% | Success |
|
||
| | ARC-Easy | Normalized Accuracy | 36.49% | Success |
|
||
| | MMLU | Accuracy | 25.64% | Success |
|
||
| | ARC-Challenge | Normalized Accuracy | 25.17% | Success |
|
||
| | OpenBookQA | Normalized Accuracy | 25.40% | Success |
|
||
| **Language Modeling** | LAMBADA | Accuracy | 15.87% | Success |
|
||
| | WikiText-2 | Word Perplexity | 251.76 | Success |
|
||
|
||
*Note: The Arithmetic benchmark failed due to outdated script support (`arithmetic.py`), and SocialIQA failed due to a registration tag error (`siqa`). Total baseline execution completed successfully for all other 15 tasks.*
|
||
|
||
|
||
## Uses
|
||
|
||
### Direct Use
|
||
The model can be used via the 🤗 Transformers library for standard text generation.
|
||
It expects chat‑formatted input (see example below).
|
||
|
||
### Downstream Use
|
||
Because of the open Apache‑2.0 license, you may fine‑tune Quark-50m‑Instruct on your own data for
|
||
domain‑specific tasks – for instance, a customer‑support bot, a code reviewer, or a story writer.
|
||
|
||
### Limitations
|
||
- Limited world knowledge (stopped at mid‑2025 pretraining data).
|
||
- Short context window (2,048 tokens).
|
||
- Small size means it can make more factual mistakes than larger models.
|
||
|
||
## How to Get Started
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
||
model_name = "ThingAI/Quark-50m-Instruct"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
|
||
|
||
messages = [
|
||
{"role": "system", "content": "You are Quark, a helpful assistant."},
|
||
{"role": "user", "content": "Explain group query attention in one sentence."}
|
||
]
|
||
|
||
inputs = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=True,
|
||
add_generation_prompt=True,
|
||
return_tensors="pt"
|
||
).to(model.device)
|
||
|
||
outputs = model.generate(inputs, max_new_tokens=128)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |