132 lines
4.8 KiB
Markdown
132 lines
4.8 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- chat
|
||
- instruct
|
||
- small-model
|
||
- 135m
|
||
- quark
|
||
---
|
||
|
||
|
||
|
||
|
||
Quark‑135M is a **135M parameter** conversational AI assistant, trained from scratch and then fine‑tuned to be **helpful, respectful, honest** and to remember a clear identity.
|
||
|
||
* **Base model:** Quark‑135M
|
||
* **Instruction tuning:** supervised fine‑tuning on a small, curated dataset of identity‑aware conversations
|
||
* **Developers:** OvercastLab and ThingsAI
|
||
* **License:** Apache‑2.0
|
||
|
||
---
|
||
|
||
## Model Architecture
|
||
|
||
The model follows a **Llama‑style decoder‑only transformer** (similar to SmolLM) with the following components:
|
||
|
||
| Component | Value |
|
||
|-------------------|----------------------|
|
||
| Vocab size | 49 152 |
|
||
| Hidden size (`d_model`) | 576 |
|
||
| Number of layers | 30 |
|
||
| Attention heads | 9 |
|
||
| KV heads (GQA) | 3 |
|
||
| Head dim | 64 |
|
||
| FFN dimension | 1 536 |
|
||
| Activation | SwiGLU |
|
||
| Normalization | RMSNorm |
|
||
| Positional encoding| Rotary Embeddings (RoPE, θ=10 000) |
|
||
| Max sequence length | 2 048 |
|
||
| Weight tying | Embedding / LM head |
|
||
|
||
**Total trainable parameters:** ~135 M
|
||
|
||
---
|
||
|
||
## Evaluation Results
|
||
|
||
The table below reports zero‑shot performance on several common benchmarks, evaluated using `lm‑eval‑harness` with `apply_chat_template=True`. All scores are shown as percentages.
|
||
|
||
| Benchmark | Metric | Score |
|
||
|---------------------|-----------|--------:|
|
||
| **HellaSwag** | acc_norm | 31.37% |
|
||
| **ARC-Easy** | acc_norm | 41.46% |
|
||
| **ARC-Challenge** | acc_norm | 25.09% |
|
||
| **PIQA** | acc_norm | 61.26% |
|
||
| **MMLU** (avg) | acc | 23.17% |
|
||
| MMLU Humanities | acc | 24.23% |
|
||
| MMLU Social Sciences| acc | 22.59% |
|
||
| MMLU STEM | acc | 22.04% |
|
||
| MMLU Other | acc | 23.27% |
|
||
| **CommonsenseQA** | acc | 20.56% |
|
||
| **OpenBookQA** | acc_norm | 27.20% |
|
||
| **Winogrande** | acc | 50.20% |
|
||
| **TriviaQA** | exact_match | 0.07% |
|
||
|
||
**Key takeaways:**
|
||
|
||
* **HellaSwag (31.37%)** is above random chance (25%) but far below models pre‑trained on hundreds of billions of tokens. This reflects the modest 15 B token pre‑training budget.
|
||
* **PIQA (61.26%)** shows the model has basic physical reasoning, benefiting from the pre‑training mix.
|
||
* **TriviaQA (0.07%)** confirms the model has **almost no factual recall** – it was not exposed to a large enough knowledge corpus.
|
||
* **MMLU (23.17%)** is near random for a 4‑option task, indicating very limited academic knowledge.
|
||
|
||
---
|
||
|
||
## Intended Use
|
||
|
||
Quark‑135M‑Instruct is a **small conversational assistant** that excels at:
|
||
|
||
- Polite, identity‑aware small talk
|
||
- Refusing gracefully when it doesn’t know something
|
||
- Following simple instructions (e.g., greetings, name recall, basic Q&A)
|
||
|
||
It is **not suitable** for tasks requiring factual accuracy, deep reasoning, or reliable knowledge retrieval.
|
||
|
||
---
|
||
|
||
## Limitations
|
||
|
||
* **Small model size** – 135M parameters are an order of magnitude smaller than current frontier models.
|
||
* **Hallucinates frequently** – when asked questions beyond simple greetings or self‑description, it may invent plausible‑sounding but incorrect answers.
|
||
* **Repetitive loops** – may occasionally repeat phrases or get stuck in loops, especially with low temperature sampling.
|
||
* **Instruction coverage** – fine‑tuned on only 1 500 identity examples; it may not handle out‑of‑domain requests gracefully.
|
||
|
||
---
|
||
|
||
## How to Use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "OvercastLab/Quark-135m-Instruct" # (replace with actual HF repo)
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
torch_dtype="auto",
|
||
device_map="auto"
|
||
)
|
||
|
||
messages = [
|
||
{"role": "system", "content": "You are Quark, a helpful, respectful and honest AI assistant created by OvercastLab and ThingsAI together with Mich. Always answer as helpfully and accurately as possible."},
|
||
{"role": "user", "content": "Hi, what's your name?"}
|
||
]
|
||
|
||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
||
output_ids = model.generate(
|
||
**inputs,
|
||
max_new_tokens=150,
|
||
do_sample=True,
|
||
temperature=0.2,
|
||
top_k=50,
|
||
top_p=0.95,
|
||
repetition_penalty=1.3,
|
||
eos_token_id=tokenizer.convert_tokens_to_ids(["<|user|>", "<|system|>"]) + [tokenizer.eos_token_id],
|
||
)
|
||
response = tokenizer.decode(output_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
|
||
print(response) |