promptrails/piimask-qwen2.5-0.5b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: promptrails/piimask-qwen2.5-0.5b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-25 10:11:18 +08:00
commit c6fa701670
13 changed files with 152270 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

184
README.md Normal file
View File

@@ -0,0 +1,184 @@
---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
language:
- en
- tr
pipeline_tag: text-generation
library_name: transformers
tags:
- pii
- pii-detection
- redaction
- privacy
- ner
- qwen2
---
# piimask-qwen2.5-0.5b
A compact (0.5B) model that detects **PII** in **English and Turkish** short, informal
text — chat messages, support tickets, emails, form fields. It runs fully locally: no
data leaves the machine, no API, no key.
Given a text, it returns a JSON list of entities:
```json
{"entities": [{"type": "PERSON", "value": "Ayşe Yılmaz"}, {"type": "EMAIL", "value": "ayse@firma.com"}]}
```
Entity types: `PERSON, EMAIL, PHONE, NATIONAL_ID, IBAN, CREDIT_CARD, ADDRESS, DATE_OF_BIRTH, IP_ADDRESS`.
## Intended use & limitations
**For:** a local-first, best-effort PII detector for short, informal en/tr text —
scrubbing input before a third-party LLM, cutting PII in developer logs, a
privacy-conscious local utility.
**Not for:** a sole, compliance-grade safeguard. Redaction is a *recall* problem — a
miss is a leak — and on realistic out-of-distribution text this model misses a meaningful
fraction (concentrated on names, and on free-text entities in long or formal documents;
structured Turkish addresses are recovered by the grammar below). Do **not** make it the
only barrier before you log, store, or forward regulated data (GDPR / PCI / HIPAA). Use it
as one layer in defence-in-depth.
**Scope:** English + Turkish; short messages, not long documents; the nine types above.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "promptrails/piimask-qwen2.5-0.5b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
SYSTEM = (
"You are a PII detection engine. Extract every piece of personally identifiable "
"information from the user's text.\n\n"
'Respond with ONLY a JSON object of the form:\n'
'{"entities": [{"type": "<TYPE>", "value": "<exact substring from the text>"}]}\n\n'
"Valid types: PERSON, EMAIL, PHONE, NATIONAL_ID, IBAN, CREDIT_CARD, ADDRESS, "
"DATE_OF_BIRTH, IP_ADDRESS.\n"
"Each value must be copied verbatim from the text. If the text contains no PII, "
'respond with {"entities": []}.'
)
text = "Ben Ayşe Yılmaz, kartım 4111 1111 1111 1111, mailim ayse@firma.com"
prompt = tok.apply_chat_template(
[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],
add_generation_prompt=True, tokenize=False,
)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```
Checksummable types (EMAIL / IP / IBAN / CREDIT_CARD / NATIONAL_ID) pair with a
deterministic validation pass (format + check digit — Luhn, IBAN mod-97, TCKN) to maximise
precision on well-formed values. **Turkish addresses** — which have no checksum — pair with
the anchor grammar below instead.
### Turkish addresses — add the deterministic grammar
The model learned one Turkish address format and misses **street-first** addresses
(`Barbaros Hayrettin Paşa Sokak No:158 Daire:37 Konak/Adana`). This small, order-invariant
regex grammar catches them by structure and recovers recall to ~1.0 — add its matches to
the model output. It fires only on Turkish structure, so English is untouched:
```python
import re
_TR_IL = ("Adana Adıyaman Afyonkarahisar Afyon Ağrı Amasya Ankara Antalya Artvin Aydın "
"Balıkesir Bilecik Bingöl Bitlis Bolu Burdur Bursa Çanakkale Çankırı Çorum Denizli "
"Diyarbakır Edirne Elazığ Erzincan Erzurum Eskişehir Gaziantep Antep Giresun Gümüşhane "
"Hakkari Hatay Isparta Mersin İçel İstanbul İzmir Kars Kastamonu Kayseri Kırklareli "
"Kırşehir Kocaeli Konya Kütahya Malatya Manisa Kahramanmaraş Maraş Mardin Muğla Muş "
"Nevşehir Niğde Ordu Rize Sakarya Samsun Siirt Sinop Sivas Tekirdağ Tokat Trabzon "
"Tunceli Şanlıurfa Urfa Uşak Van Yozgat Zonguldak Aksaray Bayburt Karaman Kırıkkale "
"Batman Şırnak Bartın Ardahan Iğdır Yalova Karabük Kilis Osmaniye Düzce").split()
_L = "A-Za-zçğıöşüÇĞİÖŞÜ"
_IL = re.compile(rf"(?<![{_L}])(?:" + "|".join(sorted(map(re.escape, _TR_IL), key=len, reverse=True)) + rf")(?![{_L}])")
_ANCHOR = re.compile(r"\b(mahallesi|mahalle|mah|mh|caddesi|cadde|cad|cd|sokağı|sokak|sok|sk|bulvarı|bulvar|blv|meydanı|meydan)\b\.?", re.I)
_NUM = re.compile(r"\b(no|numara|kat|daire|blok|apt|d)\b\.?\s*[:.]?\s*\d|\bno[:.]?\s*\d|\d+/\d+", re.I)
def _start(text, a):
i = a
while i > 0 and text[i - 1] == " ": i -= 1
s = i
while i > 0:
j = i
while j > 0 and (text[j - 1].isalnum() or text[j - 1] in "çğıöşüÇĞİÖŞÜ"): j -= 1
w = text[j:i]
if w and (w[0].isupper() or w[0].isdigit()):
s = j; i = j
while i > 0 and text[i - 1] == " ": i -= 1
if i > 0 and text[i - 1] in ".,:;": break
else:
break
return s
def turkish_addresses(text):
spans = []
for m in _IL.finditer(text):
w0 = max(0, m.start() - 140)
anchors = list(_ANCHOR.finditer(text[w0:m.start()]))
if not anchors: continue
st = _start(text[w0:m.start()], anchors[0].start()) + w0
if not _NUM.search(text[st:m.end()]) and "mah" not in anchors[0].group().lower(): continue
spans.append([st, m.end()])
spans.sort(key=lambda s: (s[0], -(s[1] - s[0])))
out = []
for s in spans:
if out and s[0] < out[-1][1]:
out[-1][1] = max(out[-1][1], s[1])
else:
out.append(s)
return [text[a:b] for a, b in out]
# merge: add addresses the model missed
# seen = {e["value"] for e in entities}
# entities += [{"type": "ADDRESS", "value": v} for v in turkish_addresses(text) if v not in seen]
```
## Evaluation
Entity-level micro-F1, **full piimask** (the model *plus* the deterministic passes above)
versus [Microsoft Presidio](https://github.com/microsoft/presidio) and the ai4privacy
DeBERTa PII NER (`Isotonic/deberta-v3-base_finetuned_ai4privacy_v2`).
**Independent bilingual benchmark** — a held-out, independently-constructed set of 400
short en/tr documents (some code-switched) with adversarial distractors (product codes,
reference numbers designed to trigger false positives):
| System | micro-F1 |
|---|---|
| **piimask (0.5B)** | **0.81** |
| ai4privacy DeBERTa | 0.65 |
| Microsoft Presidio | 0.51 |
**Turkish addresses** are the clearest win. The raw model — like both baselines — learned
one address format and misses *street-first* Turkish addresses, recall **0.16**; the
deterministic grammar above is order-invariant and lifts it to **~1.0**, with no effect on
English. Presidio and DeBERTa score ~0.00 on Turkish addresses.
**Other sets:**
| Eval set | piimask | Presidio | DeBERTa |
|---|---|---|---|
| Realistic Turkish PII (held-out test) | **0.89** | 0.42 | 0.47 |
| [piimb](https://huggingface.co/datasets/piimb/pii-masking-benchmark) aggregate (en) | **0.65** | 0.55 | — |
| [gretel](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual) finance docs (en, out-of-distribution) | **0.49** | 0.37 | 0.37 |
Read these honestly: the Turkish and bilingual numbers are on piimask's target domain
(short, informal text); the gretel English row is long, formal finance documents — fully
out-of-distribution and the most conservative, like-for-like number. Redaction is a recall
problem, and on long formal text the model still misses a meaningful fraction — so use
piimask in defence-in-depth, not standing alone. DeBERTa is not scored on piimb, which
overlaps its own training data.
## License
Apache-2.0, inheriting the base model
[Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
(Apache-2.0).

24
added_tokens.json Normal file
View File

@@ -0,0 +1,24 @@
{
"</tool_call>": 151658,
"<tool_call>": 151657,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

54
chat_template.jinja Normal file
View File

@@ -0,0 +1,54 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}

27
config.json Normal file
View File

@@ -0,0 +1,27 @@
{
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"hidden_act": "silu",
"hidden_size": 896,
"initializer_range": 0.02,
"intermediate_size": 4864,
"max_position_embeddings": 32768,
"max_window_layers": 21,
"model_type": "qwen2",
"num_attention_heads": 14,
"num_hidden_layers": 24,
"num_key_value_heads": 2,
"rms_norm_eps": 1e-06,
"rope_theta": 1000000.0,
"sliding_window": 32768,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers_version": "4.43.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

14
generation_config.json Normal file
View File

@@ -0,0 +1,14 @@
{
"bos_token_id": 151643,
"pad_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"repetition_penalty": 1.1,
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20,
"transformers_version": "4.37.0"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c32ac823be0b387d7bb17a18a902f43d59f58aafcf2131836f5b7e2d2b3dc971
size 988097730

298
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,298 @@
{
"metadata": {
"total_size": 988065536,
"total_parameters": 494032768
},
"weight_map": {
"model.embed_tokens.weight": "model.safetensors",
"model.layers.0.input_layernorm.weight": "model.safetensors",
"model.layers.0.mlp.down_proj.weight": "model.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model.safetensors",
"model.layers.0.mlp.up_proj.weight": "model.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model.safetensors",
"model.layers.0.self_attn.k_proj.bias": "model.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model.safetensors",
"model.layers.0.self_attn.q_proj.bias": "model.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model.safetensors",
"model.layers.0.self_attn.v_proj.bias": "model.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model.safetensors",
"model.layers.1.input_layernorm.weight": "model.safetensors",
"model.layers.1.mlp.down_proj.weight": "model.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model.safetensors",
"model.layers.1.mlp.up_proj.weight": "model.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model.safetensors",
"model.layers.1.self_attn.k_proj.bias": "model.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model.safetensors",
"model.layers.1.self_attn.q_proj.bias": "model.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model.safetensors",
"model.layers.1.self_attn.v_proj.bias": "model.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model.safetensors",
"model.layers.10.input_layernorm.weight": "model.safetensors",
"model.layers.10.mlp.down_proj.weight": "model.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model.safetensors",
"model.layers.10.mlp.up_proj.weight": "model.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model.safetensors",
"model.layers.10.self_attn.k_proj.bias": "model.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model.safetensors",
"model.layers.10.self_attn.q_proj.bias": "model.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model.safetensors",
"model.layers.10.self_attn.v_proj.bias": "model.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model.safetensors",
"model.layers.11.input_layernorm.weight": "model.safetensors",
"model.layers.11.mlp.down_proj.weight": "model.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model.safetensors",
"model.layers.11.mlp.up_proj.weight": "model.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model.safetensors",
"model.layers.11.self_attn.k_proj.bias": "model.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model.safetensors",
"model.layers.11.self_attn.q_proj.bias": "model.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model.safetensors",
"model.layers.11.self_attn.v_proj.bias": "model.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model.safetensors",
"model.layers.12.input_layernorm.weight": "model.safetensors",
"model.layers.12.mlp.down_proj.weight": "model.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model.safetensors",
"model.layers.12.mlp.up_proj.weight": "model.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model.safetensors",
"model.layers.12.self_attn.k_proj.bias": "model.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model.safetensors",
"model.layers.12.self_attn.q_proj.bias": "model.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model.safetensors",
"model.layers.12.self_attn.v_proj.bias": "model.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model.safetensors",
"model.layers.13.input_layernorm.weight": "model.safetensors",
"model.layers.13.mlp.down_proj.weight": "model.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model.safetensors",
"model.layers.13.mlp.up_proj.weight": "model.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model.safetensors",
"model.layers.13.self_attn.k_proj.bias": "model.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model.safetensors",
"model.layers.13.self_attn.q_proj.bias": "model.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model.safetensors",
"model.layers.13.self_attn.v_proj.bias": "model.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model.safetensors",
"model.layers.14.input_layernorm.weight": "model.safetensors",
"model.layers.14.mlp.down_proj.weight": "model.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model.safetensors",
"model.layers.14.mlp.up_proj.weight": "model.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model.safetensors",
"model.layers.14.self_attn.k_proj.bias": "model.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model.safetensors",
"model.layers.14.self_attn.q_proj.bias": "model.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model.safetensors",
"model.layers.14.self_attn.v_proj.bias": "model.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model.safetensors",
"model.layers.15.input_layernorm.weight": "model.safetensors",
"model.layers.15.mlp.down_proj.weight": "model.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model.safetensors",
"model.layers.15.mlp.up_proj.weight": "model.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model.safetensors",
"model.layers.15.self_attn.k_proj.bias": "model.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model.safetensors",
"model.layers.15.self_attn.q_proj.bias": "model.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model.safetensors",
"model.layers.15.self_attn.v_proj.bias": "model.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model.safetensors",
"model.layers.16.input_layernorm.weight": "model.safetensors",
"model.layers.16.mlp.down_proj.weight": "model.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model.safetensors",
"model.layers.16.mlp.up_proj.weight": "model.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model.safetensors",
"model.layers.16.self_attn.k_proj.bias": "model.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model.safetensors",
"model.layers.16.self_attn.q_proj.bias": "model.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model.safetensors",
"model.layers.16.self_attn.v_proj.bias": "model.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model.safetensors",
"model.layers.17.input_layernorm.weight": "model.safetensors",
"model.layers.17.mlp.down_proj.weight": "model.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model.safetensors",
"model.layers.17.mlp.up_proj.weight": "model.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model.safetensors",
"model.layers.17.self_attn.k_proj.bias": "model.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model.safetensors",
"model.layers.17.self_attn.q_proj.bias": "model.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model.safetensors",
"model.layers.17.self_attn.v_proj.bias": "model.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model.safetensors",
"model.layers.18.input_layernorm.weight": "model.safetensors",
"model.layers.18.mlp.down_proj.weight": "model.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model.safetensors",
"model.layers.18.mlp.up_proj.weight": "model.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model.safetensors",
"model.layers.18.self_attn.k_proj.bias": "model.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model.safetensors",
"model.layers.18.self_attn.q_proj.bias": "model.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model.safetensors",
"model.layers.18.self_attn.v_proj.bias": "model.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model.safetensors",
"model.layers.19.input_layernorm.weight": "model.safetensors",
"model.layers.19.mlp.down_proj.weight": "model.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model.safetensors",
"model.layers.19.mlp.up_proj.weight": "model.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model.safetensors",
"model.layers.19.self_attn.k_proj.bias": "model.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model.safetensors",
"model.layers.19.self_attn.q_proj.bias": "model.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model.safetensors",
"model.layers.19.self_attn.v_proj.bias": "model.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model.safetensors",
"model.layers.2.input_layernorm.weight": "model.safetensors",
"model.layers.2.mlp.down_proj.weight": "model.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model.safetensors",
"model.layers.2.mlp.up_proj.weight": "model.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model.safetensors",
"model.layers.2.self_attn.k_proj.bias": "model.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model.safetensors",
"model.layers.2.self_attn.q_proj.bias": "model.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model.safetensors",
"model.layers.2.self_attn.v_proj.bias": "model.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model.safetensors",
"model.layers.20.input_layernorm.weight": "model.safetensors",
"model.layers.20.mlp.down_proj.weight": "model.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model.safetensors",
"model.layers.20.mlp.up_proj.weight": "model.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model.safetensors",
"model.layers.20.self_attn.k_proj.bias": "model.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model.safetensors",
"model.layers.20.self_attn.q_proj.bias": "model.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model.safetensors",
"model.layers.20.self_attn.v_proj.bias": "model.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model.safetensors",
"model.layers.21.input_layernorm.weight": "model.safetensors",
"model.layers.21.mlp.down_proj.weight": "model.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model.safetensors",
"model.layers.21.mlp.up_proj.weight": "model.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model.safetensors",
"model.layers.21.self_attn.k_proj.bias": "model.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model.safetensors",
"model.layers.21.self_attn.q_proj.bias": "model.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model.safetensors",
"model.layers.21.self_attn.v_proj.bias": "model.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model.safetensors",
"model.layers.22.input_layernorm.weight": "model.safetensors",
"model.layers.22.mlp.down_proj.weight": "model.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model.safetensors",
"model.layers.22.mlp.up_proj.weight": "model.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model.safetensors",
"model.layers.22.self_attn.k_proj.bias": "model.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model.safetensors",
"model.layers.22.self_attn.q_proj.bias": "model.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model.safetensors",
"model.layers.22.self_attn.v_proj.bias": "model.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model.safetensors",
"model.layers.23.input_layernorm.weight": "model.safetensors",
"model.layers.23.mlp.down_proj.weight": "model.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model.safetensors",
"model.layers.23.mlp.up_proj.weight": "model.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model.safetensors",
"model.layers.23.self_attn.k_proj.bias": "model.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model.safetensors",
"model.layers.23.self_attn.q_proj.bias": "model.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model.safetensors",
"model.layers.23.self_attn.v_proj.bias": "model.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model.safetensors",
"model.layers.3.input_layernorm.weight": "model.safetensors",
"model.layers.3.mlp.down_proj.weight": "model.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model.safetensors",
"model.layers.3.mlp.up_proj.weight": "model.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model.safetensors",
"model.layers.3.self_attn.k_proj.bias": "model.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model.safetensors",
"model.layers.3.self_attn.q_proj.bias": "model.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model.safetensors",
"model.layers.3.self_attn.v_proj.bias": "model.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model.safetensors",
"model.layers.4.input_layernorm.weight": "model.safetensors",
"model.layers.4.mlp.down_proj.weight": "model.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model.safetensors",
"model.layers.4.mlp.up_proj.weight": "model.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model.safetensors",
"model.layers.4.self_attn.k_proj.bias": "model.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model.safetensors",
"model.layers.4.self_attn.q_proj.bias": "model.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model.safetensors",
"model.layers.4.self_attn.v_proj.bias": "model.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model.safetensors",
"model.layers.5.input_layernorm.weight": "model.safetensors",
"model.layers.5.mlp.down_proj.weight": "model.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model.safetensors",
"model.layers.5.mlp.up_proj.weight": "model.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model.safetensors",
"model.layers.5.self_attn.k_proj.bias": "model.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model.safetensors",
"model.layers.5.self_attn.q_proj.bias": "model.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model.safetensors",
"model.layers.5.self_attn.v_proj.bias": "model.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model.safetensors",
"model.layers.6.input_layernorm.weight": "model.safetensors",
"model.layers.6.mlp.down_proj.weight": "model.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model.safetensors",
"model.layers.6.mlp.up_proj.weight": "model.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model.safetensors",
"model.layers.6.self_attn.k_proj.bias": "model.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model.safetensors",
"model.layers.6.self_attn.q_proj.bias": "model.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model.safetensors",
"model.layers.6.self_attn.v_proj.bias": "model.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model.safetensors",
"model.layers.7.input_layernorm.weight": "model.safetensors",
"model.layers.7.mlp.down_proj.weight": "model.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model.safetensors",
"model.layers.7.mlp.up_proj.weight": "model.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model.safetensors",
"model.layers.7.self_attn.k_proj.bias": "model.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model.safetensors",
"model.layers.7.self_attn.q_proj.bias": "model.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model.safetensors",
"model.layers.7.self_attn.v_proj.bias": "model.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model.safetensors",
"model.layers.8.input_layernorm.weight": "model.safetensors",
"model.layers.8.mlp.down_proj.weight": "model.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model.safetensors",
"model.layers.8.mlp.up_proj.weight": "model.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model.safetensors",
"model.layers.8.self_attn.k_proj.bias": "model.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model.safetensors",
"model.layers.8.self_attn.q_proj.bias": "model.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model.safetensors",
"model.layers.8.self_attn.v_proj.bias": "model.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model.safetensors",
"model.layers.9.input_layernorm.weight": "model.safetensors",
"model.layers.9.mlp.down_proj.weight": "model.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model.safetensors",
"model.layers.9.mlp.up_proj.weight": "model.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model.safetensors",
"model.layers.9.self_attn.k_proj.bias": "model.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model.safetensors",
"model.layers.9.self_attn.q_proj.bias": "model.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model.safetensors",
"model.layers.9.self_attn.v_proj.bias": "model.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model.safetensors",
"model.norm.weight": "model.safetensors"
}
}

31
special_tokens_map.json Normal file
View File

@@ -0,0 +1,31 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
size 11421896

207
tokenizer_config.json Normal file
View File

@@ -0,0 +1,207 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long