初始化项目，由ModelHub XC社区提供模型

Model: ThingAI/Quark-50m-v2 Source: Original Platform
2026-07-26 13:09:21 +08:00
commit 1db470128b
17 changed files with 326319 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/.ipynb_checkpoints/chat_template-checkpoint.jinja
+++ b/.ipynb_checkpoints/chat_template-checkpoint.jinja
@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}
--- a/.ipynb_checkpoints/config-checkpoint.json
+++ b/.ipynb_checkpoints/config-checkpoint.json
@@ -0,0 +1,14 @@
+{
+  "vocab_size": 49152,
+  "d_model": 384,
+  "n_heads": 6,
+  "n_kv_heads": 2,
+  "n_layers": 24,
+  "d_ff": 1024,
+  "head_dim": 64,
+  "max_seq_len": 2048,
+  "rope_theta": 10000.0,
+  "rms_eps": 1e-05,
+  "qkv_bias": true,
+  "dropout": 0.0
+}
--- a/.ipynb_checkpoints/tokenizer-checkpoint.json
+++ b/.ipynb_checkpoints/tokenizer-checkpoint.json
--- a/.ipynb_checkpoints/tokenizer_config-checkpoint.json
+++ b/.ipynb_checkpoints/tokenizer_config-checkpoint.json
@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "is_local": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}
--- a/README.md
+++ b/README.md
@@ -0,0 +1,141 @@
+---
+language:
+- it
+- en
+license: apache-2.0
+tags:
+- italian
+- causal-lm
+- small-language-model
+- trained-from-scratch
+- chatml
+- conversational
+pipeline_tag: text-generation
+---
+
+# ⚛ Quark-50M-v2
+
+**43.8M parameter Italian-first bilingual language model, trained from scratch by ThingAI.**
+
+Quark-50M is an ultra-compact causal language model that speaks fluent Italian. Designed as a proof-of-concept for small, efficient, Italian-centric AI.
+
+## Highlights
+
+- **43.8M parameters** — runs on any device, even CPU
+- **Italian-first** — trained on 60% Italian data (books, Wikipedia, web)
+- **ChatML format** — `<|im_start|>user`/`<|im_start|>assistant`
+- **Custom tokenizer** — 16k BPE, optimized for Italian (4.15 chars/token)
+- **Trained from scratch** — architecture, tokenizer, and training pipeline all custom
+
+## Architecture
+
+| Component | Value |
+|-----------|-------|
+| Parameters | 43.8M |
+| Vocabulary | 16,384 (BPE) |
+| Dimensions | 512 |
+| Layers | 12 |
+| Heads | 8 (4 KV heads, GQA) |
+| FFN | 1,408 (SwiGLU) |
+| Context | 2,048 tokens |
+| Normalization | RMSNorm |
+| Position | RoPE |
+| Weight Tying | Yes |
+
+## Training
+
+**Pretraining:** 5B tokens on a curated mix:
+
+| Dataset | Weight | Type |
+|---------|--------|------|
+| PleIAs/Italian-PD | 25% | 171K Italian books (public domain) |
+| FineWeb-2 Italian | 20% | Cleaned, deduplicated web |
+| Wikipedia IT | 15% | Encyclopedia |
+| Cosmopedia | 15% | Synthetic educational |
+| SmolLM-Corpus | 10% | Curated mix |
+| StarCoder Python | 8% | Code |
+| OpenWebMath | 7% | Mathematics |
+
+**SFT:** Fine-tuned on [quattro-chiacchiere](https://huggingface.co/datasets/ThingAI/quattro-chiacchiere), a synthetic Italian Q&A dataset generated with [Alembic](https://github.com/skein-labs/Alembic).
+
+## Usage
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+from transformers import PreTrainedTokenizerFast
+
+# Load
+ckpt_path = hf_hub_download("ThingAI/Quark-50M", "model.pt")
+model_py = hf_hub_download("ThingAI/Quark-50M", "model.py")
+tok_file = hf_hub_download("ThingAI/Quark-50M", "tokenizer.json")
+
+# Tokenizer
+tokenizer = PreTrainedTokenizerFast(tokenizer_file=tok_file)
+tokenizer.eos_token = "<|endoftext|>"
+
+# Model
+import importlib.util
+spec = importlib.util.spec_from_file_location("model", model_py)
+mod = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(mod)
+
+ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
+cfg = mod.ModelConfig(**ckpt["model_cfg"])
+model = mod.Quark(cfg).eval()
+model.load_state_dict(ckpt["model"])
+
+# Chat
+prompt = "<|im_start|>user\nQual è la capitale d'Italia?<|im_end|>\n<|im_start|>assistant\n"
+ids = tokenizer.encode(prompt, return_tensors="pt")
+with torch.no_grad():
+    for _ in range(100):
+        logits = model(ids)[1][:, -1, :].float()
+        nxt = logits.argmax(-1, keepdim=True)
+        if nxt.item() == tokenizer.convert_tokens_to_ids("<|im_end|>"): break
+        ids = torch.cat([ids, nxt], -1)
+print(tokenizer.decode(ids[0], skip_special_tokens=True))
+# La capitale d'Italia è Roma.
+```
+
+## Examples
+
+```
+Tu: Qual è la capitale d'Italia?
+Quark: La capitale d'Italia è Roma.
+
+Tu: Chi sei?
+Quark: Sono Quark, piacere di conoscerti.
+
+Tu: Come ti chiami?
+Quark: Mi chiamo Quark, piacere di conoscerti.
+```
+
+## Limitations
+
+- **43.8M parameters** — cannot perform complex reasoning or long-form generation
+- **Factual accuracy** — may hallucinate facts, especially on niche topics
+- **SFT dataset** — currently limited; more data will improve reliability
+- **No safety training** — not recommended for production without guardrails
+
+## Related
+
+- [Quark3Tokenizer](https://huggingface.co/ThingAI/Quark3Tokenizer) — the tokenizer
+- [quattro-chiacchiere](https://huggingface.co/datasets/ThingAI/quattro-chiacchiere) — the SFT dataset
+- [Alembic](https://github.com/skein-labs/Alembic) — the dataset distillation tool
+- [Glyph](https://huggingface.co/ThingAI/Glyph) — multi-task text classifier by ThingAI
+
+## Citation
+
+```bibtex
+@misc{quark50m,
+  author = {ThingAI},
+  title  = {Quark-50M-v2: Italian-First Small Language Model},
+  year   = {2026},
+  url    = {https://huggingface.co/ThingAI/Quark-50M}
+}
+```
+
+## License
+
+Apache 2.0
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "is_llama_config": true,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 6,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.6.2",
+  "use_cache": false,
+  "vocab_size": 49152
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    0,
+    2
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.6.2",
+  "use_cache": true
+}
--- a/model.pt
+++ b/model.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b7c2bae538e8df09945d4feb450fc6784194f243a6855408a2b5276773657ec
+size 175196411
--- a/model.py
+++ b/model.py
@@ -0,0 +1,90 @@
+"""Quark-50M model definition — standalone."""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from dataclasses import dataclass
+
+@dataclass
+class ModelConfig:
+    vocab_size: int = 16384; d_model: int = 512; n_heads: int = 8
+    n_kv_heads: int = 4; n_layers: int = 12; d_ff: int = 1408
+    head_dim: int = 64; max_seq_len: int = 2048; rope_theta: float = 10000.0
+    rms_eps: float = 1e-5; qkv_bias: bool = False; dropout: float = 0.0
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-5):
+        super().__init__(); self.eps = eps; self.scale = nn.Parameter(torch.ones(dim))
+    def forward(self, x):
+        return (x.float() * x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()).to(x.dtype) * self.scale
+
+class RotaryEmbedding(nn.Module):
+    def __init__(self, head_dim, max_seq_len, theta=10000.0):
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False); self._build(max_seq_len)
+    def _build(self, seq_len):
+        t = torch.arange(seq_len, device=self.inv_freq.device).float()
+        freqs = torch.outer(t, self.inv_freq); emb = torch.cat([freqs, freqs], dim=-1)
+        self.register_buffer("cos_cache", emb.cos()[None, None], persistent=False)
+        self.register_buffer("sin_cache", emb.sin()[None, None], persistent=False); self._max = seq_len
+    @staticmethod
+    def _rot(x):
+        x1, x2 = x.chunk(2, dim=-1); return torch.cat([-x2, x1], dim=-1)
+    def forward(self, q, k):
+        T = q.size(2)
+        if T > self._max: self._build(T)
+        c, s = self.cos_cache[:,:,:T], self.sin_cache[:,:,:T]
+        return q*c + self._rot(q)*s, k*c + self._rot(k)*s
+
+class GQA(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.n_heads, self.n_kv_heads = cfg.n_heads, cfg.n_kv_heads
+        self.n_groups, self.head_dim = cfg.n_heads // cfg.n_kv_heads, cfg.head_dim
+        self.q_proj = nn.Linear(cfg.d_model, cfg.n_heads * cfg.head_dim, bias=cfg.qkv_bias)
+        self.k_proj = nn.Linear(cfg.d_model, cfg.n_kv_heads * cfg.head_dim, bias=cfg.qkv_bias)
+        self.v_proj = nn.Linear(cfg.d_model, cfg.n_kv_heads * cfg.head_dim, bias=cfg.qkv_bias)
+        self.o_proj = nn.Linear(cfg.n_heads * cfg.head_dim, cfg.d_model, bias=False)
+        self.rope = RotaryEmbedding(cfg.head_dim, cfg.max_seq_len, cfg.rope_theta)
+    def forward(self, x):
+        B, T, _ = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        q, k = self.rope(q, k)
+        if self.n_groups > 1:
+            B_r, _, T_r, D_r = k.shape
+            k = k[:,:,None,:,:].expand(B_r, self.n_kv_heads, self.n_groups, T_r, D_r).reshape(B_r, self.n_heads, T_r, D_r)
+            v = v[:,:,None,:,:].expand(B_r, self.n_kv_heads, self.n_groups, T_r, D_r).reshape(B_r, self.n_heads, T_r, D_r)
+        return self.o_proj(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).contiguous().view(B, T, -1))
+
+class Block(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.norm_attn = RMSNorm(cfg.d_model, cfg.rms_eps); self.attn = GQA(cfg)
+        self.norm_ffn = RMSNorm(cfg.d_model, cfg.rms_eps)
+        self.gate = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
+        self.up = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
+        self.down = nn.Linear(cfg.d_ff, cfg.d_model, bias=False)
+    def forward(self, x):
+        x = x + self.attn(self.norm_attn(x))
+        h = self.norm_ffn(x); x = x + self.down(F.silu(self.gate(h)) * self.up(h))
+        return x
+
+class Quark(nn.Module):
+    def __init__(self, cfg):
+        super().__init__(); self.cfg = cfg
+        self.embed_tokens = nn.Embedding(cfg.vocab_size, cfg.d_model)
+        self.layers = nn.ModuleList([Block(cfg) for _ in range(cfg.n_layers)])
+        self.norm = RMSNorm(cfg.d_model, cfg.rms_eps)
+        self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
+        self.lm_head.weight = self.embed_tokens.weight
+    def forward(self, ids, labels=None):
+        x = self.embed_tokens(ids)
+        for layer in self.layers: x = layer(x)
+        logits = self.lm_head(self.norm(x))
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(logits.view(-1, self.cfg.vocab_size), labels.view(-1), ignore_index=-100)
+        return loss, logits
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae5952eb75980cb4d0a309de77867170af85f62c992c16db60c244ad71798cd6
+size 113367352
--- a/pytorch_model.bin
+++ b/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f123e7c62d0615acb63f6e02c1fef39ad165fedd47a48f734bbc58eb23860c8e
+size 226733547
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,8 @@
+{
+  "bos_token": "<|im_start|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "additional_special_tokens": [
+    "<|im_end|>"
+  ]
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,10 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<|im_start|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "model_max_length": 4096,
+  "chat_template": "{% for message in messages %}<|im_start|>{{ message['role'] }}\n{{ message['content'] }}<|im_end|>\n{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "add_bos_token": false,
+  "add_eos_token": false
+}
--- a/training_args.bin
+++ b/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82a7ba29d21c786c5562de6a6c7a3aa158352399f3651c4038d36d59bf4bd17e
+size 5304