Files

ModelHub XC c886f7fbd6 初始化项目，由ModelHub XC社区提供模型

Model: OvercastLab/Quark-50m
Source: Original Platform

2026-07-05 08:06:17 +08:00

4.4 KiB

Raw Permalink Blame History

language, license, tags, datasets, metrics, model-index, pipeline_tag

language

license

Quark-50m-Instruct

Quark-50m-Instruct is a small (≈56M parameters) decoder-only language model, fine-tuned for instruction following. It is built on the same architecture of “SmolLM” family and was fully pretrained on 5 billion tokens from HuggingFaceTB/smollm‑corpus.

Model type: Causal Language Model (LLaMA‑style decoder)
Architecture: GQA · SwiGLU · RMSNorm · RoPE · Weight‑tying
Pretraining tokens: 5 B
Fine‑tuning: Instruction‑tuned (details below)
Creators: OvercastLab (research & development lab for ML/AI)
Release date: 22 April 2026

Model Summary

Quark-50m-Instruct is designed to be an efficient assistant that can run on consumer GPUs (e.g., RTX 3070 with 8 GB VRAM) and even on CPU for light workloads. It is not competitive with large models on knowledge‑intensive tasks, but it excels at:

Simple conversational tasks
Code generation and explanation (Python)
Short text rewriting and summarisation
On‑device / edge inference

The architecture closely follows the efficient‑small‑LM blueprint popularised by SmolLM:

Component	Details
Vocab size	49,152
Hidden size	384
Layers	24
Attention	Grouped Query (6 Q heads, 2 KV heads)
FFN	SwiGLU with 1,024 intermediate
Position	RoPE (θ = 10,000)
Normalisation	RMSNorm (pre‑block)

Total trainable parameters: ≈48 M (with weight tying).

Benchmark Evaluation Metrics

Category	Benchmark	Metric	Score / Value	Status
Linguistics & Grammar	BLiMP	Accuracy	68.12%	Success
Commonsense & Reasoning	PIQA	Normalized Accuracy	57.83%	Success
	COPA	Accuracy	57.00%	Success
	BoolQ	Accuracy	52.17%	Success
	WinoGrande	Accuracy	47.36%	Success
	HellaSwag	Normalized Accuracy	28.49%	Success
	RACE	Accuracy	26.41%	Success
	CommonsenseQA	Accuracy	20.31%	Success
Academic & Knowledge	SciQ	Normalized Accuracy	49.00%	Success
	ARC-Easy	Normalized Accuracy	36.49%	Success
	MMLU	Accuracy	25.64%	Success
	ARC-Challenge	Normalized Accuracy	25.17%	Success
	OpenBookQA	Normalized Accuracy	25.40%	Success
Language Modeling	LAMBADA	Accuracy	15.87%	Success
	WikiText-2	Word Perplexity	251.76	Success

Note: The Arithmetic benchmark failed due to outdated script support (arithmetic.py), and SocialIQA failed due to a registration tag error (siqa). Total baseline execution completed successfully for all other 15 tasks.

Uses

Direct Use

The model can be used via the 🤗 Transformers library for standard text generation. It expects chat‑formatted input (see example below).

Downstream Use

Because of the open Apache‑2.0 license, you may fine‑tune Quark-50m‑Instruct on your own data for domain‑specific tasks – for instance, a customer‑support bot, a code reviewer, or a story writer.

Limitations

Limited world knowledge (stopped at mid‑2025 pretraining data).
Short context window (2,048 tokens).
Small size means it can make more factual mistakes than larger models.

How to Get Started

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ThingAI/Quark-50m-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

messages = [
    {"role": "system", "content": "You are Quark, a helpful assistant."},
    {"role": "user", "content": "Explain group query attention in one sentence."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

4.4 KiB Raw Permalink Blame History Unescape Escape