初始化项目，由ModelHub XC社区提供模型
Model: ZelligeAI/tessera-compressor Source: Original Platform
This commit is contained in:
ModelHub XC
37
.gitattributes
vendored
Normal file
37
.gitattributes
vendored
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
gguf/compressor-v31-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
95
README.md
Normal file
95
README.md
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
- zh
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
tags:
|
||||||
|
- reasoning-compression
|
||||||
|
- cjk
|
||||||
|
- chain-of-thought
|
||||||
|
- distillation
|
||||||
|
- qwen2.5
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# tessera-compressor
|
||||||
|
|
||||||
|
A 1.5B model that compresses English reasoning text into a telegraphic CJK/symbol register under deterministic fidelity gates. It minted the training data for [Tessera-Preview-9B](https://huggingface.co/ZelligeAI/tessera-preview-9b) and replaces the frontier-model teacher that originally produced the register: English reasoning text becomes compressed-register training data at local-inference cost, with no API key and no external dependency. Validation covered code-centric reasoning (103 held-out mixed blocks); behavior on distant domains is unmeasured.
|
||||||
|
|
||||||
|
**Paper:** [Tessera-Preview-9B: Compressed Reasoning at 18x Fewer Tokens, and What It Costs](https://zellige.ai/research/compressed-cjk-reasoning) — section 3.1 covers this compressor's design and acceptance record.
|
||||||
|
|
||||||
|
Example (real training pair, 85 to 49 tokens):
|
||||||
|
|
||||||
|
```text
|
||||||
|
EN : So the classes are: - Integer (line 32) - Boolean (line 262) - BitString (line 341)
|
||||||
|
- OctetString (line 693) ... Let me look at the base class to see if it defines __mul__:
|
||||||
|
|
||||||
|
CJK: Integer(line32),Boolean(line262),BitString(line341),OctetString(line693). 查基类是否定义__mul__:
|
||||||
|
```
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
The compressor operates on passages, not whole blocks. A reasoning block is segmented (code fences stay atomic), sentences are grouped into step-sized passages, each passage is classified as fact-dense or narrative, and the model compresses it against the tail of the chain built so far. Every model output then passes a deterministic gate: the passage's novel numbers and identifiers must survive as substrings, the output must not blow up in length, and it must not exceed a rules-only compression of the same passage in token count. A passage that fails any check falls back to the rules-only version, so a bad generation costs savings rather than gated content. The gate is lexical, not semantic: it prevents the loss of numbers and identifiers, and a judged semantic-equivalence check backed it at acceptance (below), but it does not by itself guarantee semantic preservation on arbitrary input.
|
||||||
|
|
||||||
|
## Acceptance record
|
||||||
|
|
||||||
|
Measured on 103 held-out reasoning blocks the model never trained on, under criteria fixed before evaluation:
|
||||||
|
|
||||||
|
| Criterion | Result |
|
||||||
|
| --- | --- |
|
||||||
|
| Per-passage fidelity gate (numbers and identifiers survive) | 99.0% |
|
||||||
|
| Median per-passage compression ratio (output/input tokens) | 0.716 |
|
||||||
|
| CJK adoption | 98.9% of compressed passages |
|
||||||
|
| Judged semantic equivalence | 103/103 blocks (teacher references on the same blocks: 97.1%) |
|
||||||
|
| Degenerate outputs | 0 |
|
||||||
|
| Net corpus savings (after 24% rules-only fallback) | 30.4% |
|
||||||
|
|
||||||
|
On whole thinks in downstream production use (45,202 pairs), the compressed rendering costs a median 0.58x the tokens of its English source.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- Root: merged model, standard Hugging Face format (bf16). Base: Qwen2.5-Coder-1.5B-Instruct, LoRA r=16 merged in.
|
||||||
|
- `gguf/compressor-v31-q8_0.gguf`: llama.cpp quantization, validated behaviorally (scores 4/4 on the same acceptance suite). q4_k_m showed visible drift and is not published.
|
||||||
|
- `scripts/`: the complete usage harness. No tokens or keys required anywhere.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Serve the model behind any OpenAI-compatible endpoint:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm serve ZelligeAI/tessera-compressor --port 8001
|
||||||
|
# or, CPU-friendly:
|
||||||
|
llama-server -m gguf/compressor-v31-q8_0.gguf --port 8001
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the harness:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd scripts && pip install -r requirements.txt
|
||||||
|
|
||||||
|
# compress one reasoning block from a text file
|
||||||
|
python compress.py --in think.txt --endpoint http://localhost:8001/v1
|
||||||
|
|
||||||
|
# compress a corpus: {"id": ..., "text": ...} per JSONL line
|
||||||
|
python compress.py --in blocks.jsonl --out compressed.jsonl \
|
||||||
|
--endpoint http://localhost:8001/v1
|
||||||
|
```
|
||||||
|
|
||||||
|
Output records carry the compressed text, source and output token counts, and per-block harness stats (model-accepted vs rules-fallback passage counts).
|
||||||
|
|
||||||
|
`scripts/` contents:
|
||||||
|
|
||||||
|
- `compress.py`: the driver. Segment, classify, compress per passage with chain context, gate, fall back on failure.
|
||||||
|
- `segmenting.py`: segmentation, passage grouping, fact extraction, classification, and the fidelity gate. Pure text processing.
|
||||||
|
- `tokenmax.py`: deterministic token-saving substitutions, used as the rules-only fallback and as a post-processor.
|
||||||
|
|
||||||
|
One note on token counting: the gate compares token counts under a tokenizer you choose (`--tokenizer`, default this repo). To reproduce the acceptance harness exactly, point it at the tokenizer of the model you are minting data for (the acceptance run used the Qwen3.5 target tokenizer).
|
||||||
|
|
||||||
|
Throughput on the acceptance hardware was 19.6K blocks/hour on one GPU, which makes minting compressed data cheap at any corpus size.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Apache-2.0, same as the base model.
|
||||||
BIN
banner.png
Normal file
BIN
banner.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 30 KiB |
53
chat_template.jinja
Normal file
53
chat_template.jinja
Normal file
@@ -0,0 +1,53 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- messages[0]['content'] }}
|
||||||
|
{%- else %}
|
||||||
|
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{{- '<|im_start|>' + message.role }}
|
||||||
|
{%- if message.content %}
|
||||||
|
{{- '\n' + message.content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if tool_call.function is defined %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %} {{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- message.content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- endif %}
|
||||||
62
config.json
Normal file
62
config.json
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen2ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": null,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1536,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 8960,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 32768,
|
||||||
|
"max_window_layers": 21,
|
||||||
|
"model_type": "qwen2",
|
||||||
|
"num_attention_heads": 12,
|
||||||
|
"num_hidden_layers": 28,
|
||||||
|
"num_key_value_heads": 2,
|
||||||
|
"pad_token_id": 151665,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_parameters": {
|
||||||
|
"rope_theta": 1000000.0,
|
||||||
|
"rope_type": "default"
|
||||||
|
},
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"unsloth_fixed": true,
|
||||||
|
"unsloth_version": "2026.7.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151936
|
||||||
|
}
|
||||||
14
generation_config.json
Normal file
14
generation_config.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"max_length": 32768,
|
||||||
|
"pad_token_id": 151665,
|
||||||
|
"repetition_penalty": 1.1,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.8,
|
||||||
|
"transformers_version": "5.5.0"
|
||||||
|
}
|
||||||
3
gguf/compressor-v31-q8_0.gguf
Normal file
3
gguf/compressor-v31-q8_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:ce8b7b729409c6ae35d8a81fb11fcc9c286c924ec4e7d0b4b052d742af285f4d
|
||||||
|
size 1646572480
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:5660fdaa65f175f6aafb2ebd35e7e6d24e535f0deecc949cd92cc7f7858a3524
|
||||||
|
size 3087467144
|
||||||
142
scripts/compress.py
Normal file
142
scripts/compress.py
Normal file
@@ -0,0 +1,142 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
compress.py — Compress English reasoning text into the telegraphic CJK register
|
||||||
|
using tessera-compressor behind any OpenAI-compatible endpoint (vLLM, llama.cpp
|
||||||
|
server, etc.). No API keys or HF token required; the endpoint is yours.
|
||||||
|
|
||||||
|
This is the same harness the compressor was accepted under: segment the block,
|
||||||
|
group sentences into step-sized passages, classify each passage, compress it
|
||||||
|
against the chain built so far, then run the deterministic fidelity gate. A
|
||||||
|
passage that fails the gate falls back to a rules-only compression, so a bad
|
||||||
|
model output costs savings, never content.
|
||||||
|
|
||||||
|
Serve the model first, e.g.:
|
||||||
|
vllm serve ZelligeAI/tessera-compressor --port 8001
|
||||||
|
or with the GGUF:
|
||||||
|
llama-server -m gguf/compressor-v31-q8_0.gguf --port 8001 # from the repo root
|
||||||
|
|
||||||
|
Then:
|
||||||
|
# one block from a text file
|
||||||
|
python compress.py --in think.txt --endpoint http://localhost:8001/v1
|
||||||
|
|
||||||
|
# a JSONL corpus: {"id": ..., "text": ...} per line
|
||||||
|
python compress.py --in blocks.jsonl --out compressed.jsonl \
|
||||||
|
--endpoint http://localhost:8001/v1
|
||||||
|
|
||||||
|
Token counting: the fidelity gate compares token counts under a target
|
||||||
|
tokenizer. For results matching the accepted harness, point --tokenizer at the
|
||||||
|
model you are producing training data FOR (default: the compressor's own
|
||||||
|
tokenizer, which is close but not identical to the Qwen3.5 target used in the
|
||||||
|
acceptance run).
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from openai import OpenAI
|
||||||
|
from tokenizers import Tokenizer
|
||||||
|
|
||||||
|
from segmenting import segment, group_steps, classify_passage, facts, gate
|
||||||
|
from tokenmax import _apply_subs
|
||||||
|
|
||||||
|
PASSAGE_SYSTEM = (
|
||||||
|
"你是推理压缩器。Re-notate the NEXT PASSAGE of a reasoning chain into telegraphic "
|
||||||
|
"CJK/symbol notation. Every NEW logical step, fact, number and identifier must "
|
||||||
|
"survive — unless already stated in the chain. Never restate chain content. "
|
||||||
|
"[passage=load]: step-lossless telegraphic. [passage=narr]: minimal stubs "
|
||||||
|
"(试X→否). Output only the re-notated continuation."
|
||||||
|
)
|
||||||
|
|
||||||
|
MAX_NEW_TOKENS = 512
|
||||||
|
|
||||||
|
|
||||||
|
def compress_block(text, client, model, ntok):
|
||||||
|
"""Compress one reasoning block. Returns (compressed_text, stats)."""
|
||||||
|
segs = group_steps(segment(text))
|
||||||
|
chain, seen = [], set()
|
||||||
|
stats = {"segments": len(segs), "model_ok": 0, "fallback": 0,
|
||||||
|
"narr_skipped": 0, "code": 0, "calls": 0}
|
||||||
|
|
||||||
|
for kind, s in segs:
|
||||||
|
if kind == "code":
|
||||||
|
chain.append(s)
|
||||||
|
seen |= facts(s)
|
||||||
|
stats["code"] += 1
|
||||||
|
continue
|
||||||
|
cls = classify_passage(s, seen, ntok)
|
||||||
|
novel = facts(s) - seen
|
||||||
|
rules_s, _ = _apply_subs(s)
|
||||||
|
if not rules_s.strip():
|
||||||
|
continue
|
||||||
|
tail = "\n".join(chain)[-500:] or "(start)"
|
||||||
|
stats["calls"] += 1
|
||||||
|
r = client.chat.completions.create(
|
||||||
|
model=model, temperature=0.0, max_tokens=MAX_NEW_TOKENS,
|
||||||
|
messages=[
|
||||||
|
{"role": "system", "content": PASSAGE_SYSTEM},
|
||||||
|
{"role": "user", "content": f"[passage={cls}]\n链:\n{tail}\n\n段:\n{s[:2000]}"},
|
||||||
|
],
|
||||||
|
extra_body={"repetition_penalty": 1.15},
|
||||||
|
)
|
||||||
|
out = (r.choices[0].message.content or "").strip()
|
||||||
|
|
||||||
|
if out == "∅" and cls == "narr" and not novel:
|
||||||
|
stats["narr_skipped"] += 1
|
||||||
|
seen |= facts(s)
|
||||||
|
continue
|
||||||
|
if gate(s, rules_s, out, ntok, novel=novel) is None:
|
||||||
|
chain.append(out)
|
||||||
|
stats["model_ok"] += 1
|
||||||
|
else:
|
||||||
|
chain.append(rules_s)
|
||||||
|
stats["fallback"] += 1
|
||||||
|
seen |= facts(s)
|
||||||
|
|
||||||
|
return "\n".join(chain), stats
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__,
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("--in", dest="inp", required=True,
|
||||||
|
help=".txt (one block) or .jsonl ({'id','text'} per line)")
|
||||||
|
ap.add_argument("--out", default=None, help="output JSONL (default: stdout)")
|
||||||
|
ap.add_argument("--endpoint", default="http://localhost:8001/v1")
|
||||||
|
ap.add_argument("--model", default="ZelligeAI/tessera-compressor",
|
||||||
|
help="served model name at the endpoint")
|
||||||
|
ap.add_argument("--tokenizer", default="ZelligeAI/tessera-compressor",
|
||||||
|
help="HF repo id or local tokenizer.json for gate token counts")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if args.tokenizer.endswith(".json"):
|
||||||
|
tok = Tokenizer.from_file(args.tokenizer)
|
||||||
|
else:
|
||||||
|
tok = Tokenizer.from_pretrained(args.tokenizer)
|
||||||
|
|
||||||
|
def ntok(s):
|
||||||
|
return len(tok.encode(s).ids) if s else 0
|
||||||
|
|
||||||
|
client = OpenAI(base_url=args.endpoint, api_key="none")
|
||||||
|
|
||||||
|
if args.inp.endswith(".jsonl"):
|
||||||
|
rows = [json.loads(l) for l in open(args.inp) if l.strip()]
|
||||||
|
else:
|
||||||
|
rows = [{"id": args.inp, "text": open(args.inp).read()}]
|
||||||
|
|
||||||
|
sink = open(args.out, "w") if args.out else sys.stdout
|
||||||
|
for row in rows:
|
||||||
|
compressed, stats = compress_block(row["text"], client, args.model, ntok)
|
||||||
|
rec = {"id": row.get("id"), "compressed": compressed,
|
||||||
|
"src_tokens": ntok(row["text"]), "out_tokens": ntok(compressed),
|
||||||
|
"harness": stats}
|
||||||
|
sink.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||||
|
sink.flush()
|
||||||
|
print(f"[{row.get('id')}] {rec['src_tokens']} -> {rec['out_tokens']} tokens "
|
||||||
|
f"(model_ok={stats['model_ok']} fallback={stats['fallback']})",
|
||||||
|
file=sys.stderr)
|
||||||
|
if args.out:
|
||||||
|
sink.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
3
scripts/requirements.txt
Normal file
3
scripts/requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
openai>=1.0
|
||||||
|
tokenizers>=0.15
|
||||||
|
transformers>=4.40 # tokenmax.py standalone CLI only
|
||||||
121
scripts/segmenting.py
Normal file
121
scripts/segmenting.py
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
"""
|
||||||
|
segmenting.py — Passage segmentation, classification, and fidelity gates for the
|
||||||
|
tessera-compressor harness.
|
||||||
|
|
||||||
|
Extracted from the harness the compressor was accepted under (same functions the
|
||||||
|
teacher mint used). Pure text processing: no network, no credentials.
|
||||||
|
|
||||||
|
Flow: segment -> group_steps -> classify_passage per passage -> model call ->
|
||||||
|
gate -> rules fallback on failure. A failed passage costs a few dozen tokens of
|
||||||
|
savings, never content.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
|
||||||
|
CJK = re.compile(r'[一-鿿㐀-䶿]')
|
||||||
|
NUM = re.compile(r'\d+(?:\.\d+)?')
|
||||||
|
IDENT = re.compile(r'`[^`\n]+`|\b[A-Za-z]+(?:_[A-Za-z0-9]+)+\b|\b[a-z]+[A-Z][A-Za-z0-9]*\b')
|
||||||
|
FENCE = re.compile(r'```.*?```', re.DOTALL)
|
||||||
|
SENT_SPLIT = re.compile(r'(?<=[.!?;])\s+')
|
||||||
|
_LIST_MARKER = re.compile(r'(?:^|[\n\s(])(\d{1,2})[.)]\s')
|
||||||
|
_OPS = set('+-*/=<>≤≥≠∈∀∃¬→⇒%^{}[]')
|
||||||
|
|
||||||
|
|
||||||
|
def segment(text):
|
||||||
|
"""Split a reasoning block into ordered segments; code fences are atomic and marked."""
|
||||||
|
segs = [] # (kind, text) kind ∈ {'code','prose'}
|
||||||
|
pos = 0
|
||||||
|
for m in FENCE.finditer(text):
|
||||||
|
before = text[pos:m.start()]
|
||||||
|
segs.extend(('prose', s) for s in _split_prose(before))
|
||||||
|
segs.append(('code', m.group(0)))
|
||||||
|
pos = m.end()
|
||||||
|
segs.extend(('prose', s) for s in _split_prose(text[pos:]))
|
||||||
|
return [(k, s) for k, s in segs if s.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _split_prose(text):
|
||||||
|
out = []
|
||||||
|
for line in text.split('\n'):
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
out.extend(s.strip() for s in SENT_SPLIT.split(line) if s.strip())
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def group_steps(segs, max_words=160, max_sents=10):
|
||||||
|
"""Merge consecutive prose sentences into step-sized passages; code stays atomic."""
|
||||||
|
out, buf, words = [], [], 0
|
||||||
|
|
||||||
|
def flush():
|
||||||
|
nonlocal buf, words
|
||||||
|
if buf:
|
||||||
|
out.append(('prose', ' '.join(buf)))
|
||||||
|
buf, words = [], 0
|
||||||
|
|
||||||
|
for kind, s in segs:
|
||||||
|
if kind == 'code':
|
||||||
|
flush()
|
||||||
|
out.append((kind, s))
|
||||||
|
continue
|
||||||
|
buf.append(s)
|
||||||
|
words += len(s.split())
|
||||||
|
if words >= max_words or len(buf) >= max_sents:
|
||||||
|
flush()
|
||||||
|
flush()
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def facts(s):
|
||||||
|
"""Numbers + identifiers that must survive compression.
|
||||||
|
List-enumeration markers ("1. Load...") are structure, not facts."""
|
||||||
|
nums = set(NUM.findall(s)) - set(_LIST_MARKER.findall(s))
|
||||||
|
idents = set(i.strip('`') for i in IDENT.findall(s))
|
||||||
|
return nums | idents
|
||||||
|
|
||||||
|
|
||||||
|
def facts_preserved(src, out):
|
||||||
|
"""Substring presence — regex \\b breaks against adjacent CJK chars.
|
||||||
|
Returns the list of MISSING facts (empty list = all preserved)."""
|
||||||
|
out_n = out.replace(',', '')
|
||||||
|
return [f for f in facts(src) if f.replace(',', '') not in out_n]
|
||||||
|
|
||||||
|
|
||||||
|
def classify_passage(seg, seen_facts, ntok):
|
||||||
|
"""'load' = fact-dense or novel-fact-bearing (step-faithful treatment);
|
||||||
|
'narr' = search/narrative (stub treatment).
|
||||||
|
ntok is a callable: text -> token count under your target tokenizer."""
|
||||||
|
f = facts(seg)
|
||||||
|
novel = f - seen_facts
|
||||||
|
toks = max(ntok(seg), 1)
|
||||||
|
dens = (len(NUM.findall(seg)) + len(IDENT.findall(seg))
|
||||||
|
+ sum(seg.count(o) for o in _OPS)) / toks
|
||||||
|
if novel and (dens >= 0.08 or len(novel) >= 3):
|
||||||
|
return 'load'
|
||||||
|
if dens >= 0.15:
|
||||||
|
return 'load'
|
||||||
|
return 'narr'
|
||||||
|
|
||||||
|
|
||||||
|
def gate(src_seg, rules_seg, out, ntok, novel=None):
|
||||||
|
"""Deterministic per-passage fidelity gate.
|
||||||
|
Returns None if the model output is admissible, else a short fail-reason
|
||||||
|
string; on failure the caller uses rules_seg instead.
|
||||||
|
|
||||||
|
novel: the passage's facts that are NOT already in the accumulated chain.
|
||||||
|
The prompt tells the model never to restate chain content, so only novel
|
||||||
|
facts are required to survive (matching the acceptance harness). Pass None
|
||||||
|
to require every fact of the passage (stricter, for chainless use)."""
|
||||||
|
if not out or not out.strip():
|
||||||
|
return "empty"
|
||||||
|
if '```' in out:
|
||||||
|
return "fence"
|
||||||
|
if len(out) > 2 * len(src_seg) + 40: # explanation/blow-up guard
|
||||||
|
return "blowup"
|
||||||
|
required = facts(src_seg) if novel is None else novel
|
||||||
|
out_n = out.replace(',', '')
|
||||||
|
if any(f.replace(',', '') not in out_n for f in required):
|
||||||
|
return "facts"
|
||||||
|
if ntok(out) > ntok(rules_seg): # must not exceed the rules-only version
|
||||||
|
return "tokens"
|
||||||
|
return None
|
||||||
431
scripts/tokenmax.py
Normal file
431
scripts/tokenmax.py
Normal file
@@ -0,0 +1,431 @@
|
|||||||
|
"""
|
||||||
|
tokenmax.py — Deterministic token-maxing post-processor for compressed think blocks.
|
||||||
|
|
||||||
|
Applies ONLY substitutions that are verified to save tokens on the Qwen 248K tokenizer
|
||||||
|
(OmniCoder-9B / Qwen3.5). Every substitution was tested in-context (not isolation) to
|
||||||
|
confirm real token savings without boundary interference.
|
||||||
|
|
||||||
|
Design:
|
||||||
|
- LLM does semantic compression (what to keep vs drop)
|
||||||
|
- This code enforces consistent notation deterministically
|
||||||
|
- GUARD: only returns the processed version if ntok(result) < ntok(original)
|
||||||
|
- Idempotent: safe to run multiple times
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
from caveman.compress.tokenmax import tokenmax, tokenmax_with_stats
|
||||||
|
compressed = tokenmax(think_text, tokenizer)
|
||||||
|
compressed, stats = tokenmax_with_stats(think_text, tokenizer)
|
||||||
|
|
||||||
|
Verified: 2026-05-31 on Qwen 248K vocab. 27/28 substitutions save in-context.
|
||||||
|
Zero false positives. One zero-effect (贪心 for "greedy" — boundary-dependent).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
# ── Phase 1: Filler drops ──────────────────────────────────────────────
|
||||||
|
# Phrases that carry zero information in compressed reasoning.
|
||||||
|
# Only patterns that are NEVER load-bearing in a think block.
|
||||||
|
_FILLER_PATTERNS = [
|
||||||
|
# Metacognition (the model narrating its own process)
|
||||||
|
r"\bI need to\b",
|
||||||
|
r"\bwe need to\b",
|
||||||
|
r"\bI will\b",
|
||||||
|
r"\bI'll\b",
|
||||||
|
r"\blet me\b",
|
||||||
|
r"\blet's\b",
|
||||||
|
r"\bI want to\b",
|
||||||
|
r"\bI should\b",
|
||||||
|
r"\bwe should\b",
|
||||||
|
# Hedging
|
||||||
|
r"\bprobably\b",
|
||||||
|
r"\bbasically\b",
|
||||||
|
r"\bessentially\b",
|
||||||
|
r"\bit seems like\b",
|
||||||
|
# Filler transitions
|
||||||
|
r"\bin order to\b",
|
||||||
|
r"\bfirst of all\b",
|
||||||
|
r"\bin other words\b",
|
||||||
|
r"\bon the other hand\b",
|
||||||
|
r"\bmore specifically\b",
|
||||||
|
r"\bto be more precise\b",
|
||||||
|
r"\band so on\b",
|
||||||
|
# Conversational padding (require word boundary at end to avoid "Greatest", "Perfectly")
|
||||||
|
r"\bGreat\b[,!.]?\s*",
|
||||||
|
r"\bPerfect\b[,!.]?\s*",
|
||||||
|
# Obvious statements
|
||||||
|
r"\bAs (?:we|you) can see\b",
|
||||||
|
r"\bAs mentioned (?:above|earlier|before)\b",
|
||||||
|
]
|
||||||
|
|
||||||
|
# ── Phase 2: Phrase → cheapest token ───────────────────────────────────
|
||||||
|
# Ordered LONGEST FIRST to prevent partial matches.
|
||||||
|
# Each entry: (regex_pattern, replacement, category)
|
||||||
|
# Categories: 'cjk', 'symbol', 'abbrev' — for stats tracking.
|
||||||
|
_SUBSTITUTIONS = [
|
||||||
|
# ── COMPOUND PATTERNS FIRST (must fire before their components) ──
|
||||||
|
|
||||||
|
# Verbose comparison phrases (+5t savings)
|
||||||
|
(r'\bis\s+greater\s+than\s+or\s+equal\s+to\b', '≥', 'symbol'), # +5t
|
||||||
|
(r'\bis\s+less\s+than\s+or\s+equal\s+to\b', '≤', 'symbol'), # +5t
|
||||||
|
|
||||||
|
# Verbose discourse phrases (+3t savings)
|
||||||
|
(r'\bwe\s+can\s+see\s+that\b', '可知', 'cjk'), # +3t
|
||||||
|
(r'\bat\s+the\s+same\s+time\b', '同时', 'cjk'), # +3t
|
||||||
|
(r'\bthat\s+is\s+to\s+say\b', '即', 'cjk'), # +3t
|
||||||
|
|
||||||
|
# Multi-word phrases (+2t savings)
|
||||||
|
(r'\bin\s+this\s+case\b', '此时', 'cjk'), # +2t
|
||||||
|
(r'\bthe\s+number\s+of\b', '个数', 'cjk'), # +2t
|
||||||
|
(r'\bis\s+equal\s+to\b', '等于', 'cjk'), # +2t
|
||||||
|
|
||||||
|
# Complexity boilerplate (biggest per-occurrence savings)
|
||||||
|
(r'[Oo]\(n\)\s*time[,;]?\s*[Oo]\(n\)\s*space\.?', 'O(n|n).', 'abbrev'),
|
||||||
|
(r'[Oo]\(n\)\s*time[,;]?\s*[Oo]\(1\)\s*space\.?', 'O(n|1).', 'abbrev'),
|
||||||
|
(r'[Oo]\(n\s*log\s*n\)\s*time[,;]?\s*[Oo]\(n\)\s*space', 'O(n㏒n|n)', 'abbrev'),
|
||||||
|
(r'[Oo]\(n\s*log\s*n\)\s*time[,;]?\s*[Oo]\(1\)\s*space', 'O(n㏒n|1)', 'abbrev'),
|
||||||
|
(r'[Tt]ime\s*complexity[:\s]+', 'T=', 'abbrev'),
|
||||||
|
(r'[Ss]pace\s*complexity[:\s]+', 'S=', 'abbrev'),
|
||||||
|
|
||||||
|
# Multi-word compounds (BEFORE their single-word components)
|
||||||
|
(r'\bassume without loss of generality\b', '设 不妨', 'cjk'), # before "assume"
|
||||||
|
(r'\bproof by contradiction\b', '反证', 'cjk'), # before "proof", "contradiction"
|
||||||
|
(r'\bnecessary and sufficient\b', '充要', 'cjk'), # before "sufficient"
|
||||||
|
(r'\bnot equal(?:\s+to)?\b', '≠', 'symbol'), # before "is not", "does not"
|
||||||
|
(r'\bif and only if\b', 'iff', 'abbrev'), # before "for all"
|
||||||
|
(r'\bmuch greater than\b', '≫', 'symbol'), # before "greater than"
|
||||||
|
(r'\bkeep track(?:\s+of)?\b', '记录', 'cjk'), # before article strip
|
||||||
|
(r'\bin ascending order\b', 'asc', 'abbrev'),
|
||||||
|
(r'\bin descending order\b', 'desc', 'abbrev'),
|
||||||
|
(r'\bmaximum value\b', '最大值', 'cjk'),
|
||||||
|
(r'\bminimum value\b', '最小值', 'cjk'),
|
||||||
|
(r'\breturn value\b', '返回値', 'cjk'),
|
||||||
|
(r'\brather than\b', '而非', 'cjk'),
|
||||||
|
(r'\baccording to\b', '按照', 'cjk'),
|
||||||
|
|
||||||
|
# DS compounds (before components)
|
||||||
|
(r'\bdoubly linked list\b', 'DLL', 'abbrev'), # BEFORE "linked list"
|
||||||
|
(r'\bbinary indexed tree\b', 'BIT', 'abbrev'), # BEFORE "binary"
|
||||||
|
(r'\bminimum spanning tree\b', 'MST', 'abbrev'),
|
||||||
|
(r'\bdepth[- ]first search\b', 'DFS', 'abbrev'),
|
||||||
|
(r'\bbreadth[- ]first search\b', 'BFS', 'abbrev'),
|
||||||
|
(r'\bdynamic programming\b', 'DP', 'abbrev'),
|
||||||
|
(r'\bdivide and conquer\b', '分治', 'cjk'),
|
||||||
|
(r'\bmonot(?:onic|one)\s*stack\b', '单调栈', 'cjk'),
|
||||||
|
(r'\btime limit exceeded\b', '超时', 'cjk'),
|
||||||
|
(r'\bout of bounds\b', '越界', 'cjk'),
|
||||||
|
(r'\bremove duplicates?\b', '去重', 'cjk'),
|
||||||
|
(r'\benumerate all\b', '穷举', 'cjk'),
|
||||||
|
(r'\bbinary search\b', '二分', 'cjk'),
|
||||||
|
(r'\bsliding window\b', 'sw', 'abbrev'),
|
||||||
|
(r'\bunion[- ]find\b', 'UF', 'abbrev'),
|
||||||
|
(r'\btopological sort\b', '拓扑序', 'cjk'),
|
||||||
|
(r'\bshortest path\b', 'sp', 'abbrev'),
|
||||||
|
(r'\blinked list\b', 'LL', 'abbrev'),
|
||||||
|
(r'\bpriority queue\b', 'heap', 'abbrev'),
|
||||||
|
(r'\bprefix sum\b', 'ps', 'abbrev'),
|
||||||
|
(r'\bbrute force\b', '暴力', 'cjk'),
|
||||||
|
(r'\bno solution\b', '无解', 'cjk'),
|
||||||
|
(r'\bedge cases?\b', '边界', 'cjk'),
|
||||||
|
(r'\bbase case\b', 'bc', 'abbrev'),
|
||||||
|
(r'\bworst case\b', 'wc', 'abbrev'),
|
||||||
|
|
||||||
|
# ── SINGLE-WORD SUBSTITUTIONS (safe after compounds consumed) ──
|
||||||
|
|
||||||
|
# +4t savings
|
||||||
|
(r'\bobviously\b', '显然', 'cjk'),
|
||||||
|
# +3t savings
|
||||||
|
(r'\bredundant\b', '冗余', 'cjk'),
|
||||||
|
(r'\bsatisf(?:y|ies|ied)\b', '满足', 'cjk'), # +2t, 333x in data
|
||||||
|
(r'\bunsorted\b', '无序', 'cjk'),
|
||||||
|
(r'\bdue to\b', '由于', 'cjk'),
|
||||||
|
(r'\bhence\b', '故', 'cjk'),
|
||||||
|
(r'\bnamely\b', '即', 'cjk'),
|
||||||
|
(r'\bassume\b', '设', 'cjk'),
|
||||||
|
(r'\bsuppose\b', '设', 'cjk'),
|
||||||
|
(r'\bderive\b', '推导', 'cjk'),
|
||||||
|
# +2t savings
|
||||||
|
(r'\bmonotone\b', '单调', 'cjk'),
|
||||||
|
(r'\bconvergent\b', '收敛', 'cjk'),
|
||||||
|
(r'\bdivergent\b', '发散', 'cjk'),
|
||||||
|
(r'\bcommutative\b', '交换', 'cjk'),
|
||||||
|
(r'\bdeterministic\b', '确定', 'cjk'),
|
||||||
|
(r'\bprobabilistic\b', '概率', 'cjk'),
|
||||||
|
(r'\bprove\b', '证明', 'cjk'),
|
||||||
|
(r'\bproof\b', '证明', 'cjk'),
|
||||||
|
(r'\bflip\b', '翻转', 'cjk'),
|
||||||
|
(r'\bsorted\b(?!\s*[=(\[])', '有序', 'cjk'), # not before = ( [ (assignment/call)
|
||||||
|
# +1t savings
|
||||||
|
(r'\bbacktrack(?:ing)?\b', '回溯', 'cjk'),
|
||||||
|
(r'\btravers(?:e|al|ing)\b', '遍历', 'cjk'),
|
||||||
|
(r'\brecursi(?:on|ve|vely)\b', '递归', 'cjk'),
|
||||||
|
(r'\bcontradiction\b', '矛盾', 'cjk'),
|
||||||
|
(r'\bsufficient\b', '充分', 'cjk'),
|
||||||
|
(r'\bequivalent\b', '等价', 'cjk'),
|
||||||
|
(r'\bsymmetric\b', '对称', 'cjk'),
|
||||||
|
(r'\binvariant\b', '不变', 'cjk'),
|
||||||
|
(r'\bexponential\b', '指数', 'cjk'),
|
||||||
|
(r'\bpermutation\b', '排列', 'cjk'),
|
||||||
|
(r'\badjacent\b', '相邻', 'cjk'),
|
||||||
|
(r'\boptimal\b', '最优', 'cjk'),
|
||||||
|
(r'\bfeasible\b', '可行', 'cjk'),
|
||||||
|
(r'\binduction\b', '归纳', 'cjk'),
|
||||||
|
(r'\bmaintain\b', '维护', 'cjk'),
|
||||||
|
(r'\bswap\b(?!\s*[=(\[])', '交换', 'cjk'), # not before = ( [ (assignment/call)
|
||||||
|
(r'\bcumulative\b', '累积', 'cjk'),
|
||||||
|
(r'\bquotient\b', '商', 'cjk'),
|
||||||
|
(r'\bmemoiz(?:ation|e)\b', 'memo', 'abbrev'),
|
||||||
|
# +2t savings (mined from v19 data)
|
||||||
|
(r'\bmathematical\b', '数学', 'cjk'),
|
||||||
|
(r'\bcorresponding(?:ly)?\b', '对应', 'cjk'),
|
||||||
|
(r'\brequirement\b', '需求', 'cjk'),
|
||||||
|
# +1t savings (mined from v19 data)
|
||||||
|
(r'\bcomplexity\b', '复杂度', 'cjk'),
|
||||||
|
(r'\bsimilarly\b', '同理', 'cjk'),
|
||||||
|
(r'\bsubstitut(?:e|ion)\b', '代入', 'cjk'),
|
||||||
|
(r'\bincreasing(?:ly)?\b', '递增', 'cjk'),
|
||||||
|
(r'\bdecreasing(?:ly)?\b', '递减', 'cjk'),
|
||||||
|
(r'\brespectively\b', '分别', 'cjk'),
|
||||||
|
(r'\bnecessarily\b', '必然', 'cjk'),
|
||||||
|
(r'\btransformation\b', '变换', 'cjk'),
|
||||||
|
(r'\bprerequisite\b', '前提', 'cjk'),
|
||||||
|
(r'\bconsequently\b', '从而', 'cjk'),
|
||||||
|
(r'\boverlapping\b', '重叠', 'cjk'),
|
||||||
|
(r'\bcontribut(?:e|ion)\b', '贡献', 'cjk'),
|
||||||
|
(r'\bindependent(?:ly)?\b', '独立', 'cjk'),
|
||||||
|
(r'\bimpossible\b', '不可能', 'cjk'),
|
||||||
|
(r'\biterat(?:e|ion|ing)\b', '迭代', 'cjk'),
|
||||||
|
(r'\benumerat(?:e|ion|ing)\b', '枚举', 'cjk'),
|
||||||
|
|
||||||
|
# ── LOGIC SYMBOLS ──
|
||||||
|
(r'\btherefore\b', '⇒', 'symbol'),
|
||||||
|
(r'\bthus\b', '⇒', 'symbol'),
|
||||||
|
(r'\bsuch that\b', 'st', 'abbrev'),
|
||||||
|
(r'\bthere exists?\b', '∃', 'symbol'),
|
||||||
|
(r'\bfor each\b', '∀', 'symbol'),
|
||||||
|
(r'\bfor every\b', '∀', 'symbol'),
|
||||||
|
(r'\bfor all\b', '∀', 'symbol'),
|
||||||
|
(r'\bdoes not\b', '¬', 'symbol'),
|
||||||
|
(r"\bdoesn't\b", '¬', 'symbol'),
|
||||||
|
(r'\bis not\b(?!\s+(?:None|null|undefined|empty|zero|0))', '非', 'cjk'), # protect "is not None" etc
|
||||||
|
(r'\bat least\b', '≥', 'symbol'),
|
||||||
|
(r'\bat most\b', '≤', 'symbol'),
|
||||||
|
(r'\bgreater than\b', '>', 'symbol'),
|
||||||
|
(r'\bless than\b', '<', 'symbol'),
|
||||||
|
|
||||||
|
# ── ARTICLE STRIPPING (last — lowest priority) ──
|
||||||
|
(r'\bthe\b\s+(?!(?:same|only|first|last|next|other)\b)', '', 'filler'), # protect "the same", "the only" etc
|
||||||
|
(r'\ba\b\s+(?=[bcdfghjklmnpqrstvwxyz])', '', 'filler'),
|
||||||
|
(r'\ban\b\s+', '', 'filler'),
|
||||||
|
]
|
||||||
|
|
||||||
|
# ── Compile once ───────────────────────────────────────────────────────
|
||||||
|
_FILLER_COMPILED = [(re.compile(p, re.IGNORECASE), '') for p in _FILLER_PATTERNS]
|
||||||
|
_SUBS_COMPILED = [(re.compile(p, re.IGNORECASE), r, cat) for p, r, cat in _SUBSTITUTIONS]
|
||||||
|
|
||||||
|
|
||||||
|
def _ntok(text: str, tokenizer) -> int:
|
||||||
|
"""Token count using the provided tokenizer."""
|
||||||
|
return len(tokenizer.encode(text, add_special_tokens=False))
|
||||||
|
|
||||||
|
|
||||||
|
def _protect_code_fences(text: str) -> tuple[str, list]:
|
||||||
|
"""Extract code-fenced blocks, replace with placeholders.
|
||||||
|
Returns (text_with_placeholders, list_of_extracted_blocks)."""
|
||||||
|
blocks = []
|
||||||
|
def _replace(m):
|
||||||
|
blocks.append(m.group(0))
|
||||||
|
return f'\x00CODEFENCE{len(blocks)-1}\x00'
|
||||||
|
# Match ```...``` and inline `...` (non-greedy)
|
||||||
|
protected = re.sub(r'```.*?```|`[^`\n]+`', _replace, text, flags=re.DOTALL)
|
||||||
|
return protected, blocks
|
||||||
|
|
||||||
|
|
||||||
|
def _restore_code_fences(text: str, blocks: list) -> str:
|
||||||
|
"""Restore code-fenced blocks from placeholders."""
|
||||||
|
for i, block in enumerate(blocks):
|
||||||
|
text = text.replace(f'\x00CODEFENCE{i}\x00', block)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_subs(text: str) -> tuple[str, dict]:
|
||||||
|
"""Apply all substitutions, return (result, stats).
|
||||||
|
Code fences (``` and inline `) are protected from substitution."""
|
||||||
|
stats = {'filler_drops': 0, 'cjk': 0, 'symbol': 0, 'abbrev': 0, 'total_subs': 0}
|
||||||
|
|
||||||
|
# Phase 0: protect code fences from substitution
|
||||||
|
text, code_blocks = _protect_code_fences(text)
|
||||||
|
|
||||||
|
# Phase 1: filler drops
|
||||||
|
for pat, repl in _FILLER_COMPILED:
|
||||||
|
text, n = pat.subn(repl, text)
|
||||||
|
if n:
|
||||||
|
stats['filler_drops'] += n
|
||||||
|
stats['total_subs'] += n
|
||||||
|
|
||||||
|
# Phase 2: substitutions
|
||||||
|
for pat, repl, cat in _SUBS_COMPILED:
|
||||||
|
text, n = pat.subn(repl, text)
|
||||||
|
if n:
|
||||||
|
stats[cat] = stats.get(cat, 0) + n
|
||||||
|
stats['total_subs'] += n
|
||||||
|
|
||||||
|
# Phase 3: restore code fences
|
||||||
|
text = _restore_code_fences(text, code_blocks)
|
||||||
|
|
||||||
|
# Phase 4: whitespace normalization
|
||||||
|
text = re.sub(r'[ \t]+', ' ', text)
|
||||||
|
text = re.sub(r'\n{3,}', '\n\n', text)
|
||||||
|
text = re.sub(r' *\n *', '\n', text)
|
||||||
|
text = text.strip()
|
||||||
|
|
||||||
|
return text, stats
|
||||||
|
|
||||||
|
|
||||||
|
def tokenmax(text: str, tokenizer, force_cjk: bool = False) -> str:
|
||||||
|
"""Apply token-maxing. Returns original if no savings achieved.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: The think block content (without <think> tags).
|
||||||
|
tokenizer: A HuggingFace tokenizer with .encode() method.
|
||||||
|
force_cjk: If True, always return the processed version when CJK
|
||||||
|
substitutions were applied, even if total token count increased.
|
||||||
|
Use this to maximize CJK adoption in training data.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The token-maxed text, or the original if processing didn't save tokens
|
||||||
|
(unless force_cjk=True and CJK subs were applied).
|
||||||
|
"""
|
||||||
|
if not text or not text.strip():
|
||||||
|
return text
|
||||||
|
|
||||||
|
original_tokens = _ntok(text, tokenizer)
|
||||||
|
result, stats = _apply_subs(text)
|
||||||
|
result_tokens = _ntok(result, tokenizer)
|
||||||
|
|
||||||
|
# GUARD: only return processed version if it actually saves tokens
|
||||||
|
# OVERRIDE: force_cjk bypasses the guard when CJK substitutions were made
|
||||||
|
if result_tokens < original_tokens:
|
||||||
|
return result
|
||||||
|
if force_cjk and stats.get('cjk', 0) > 0:
|
||||||
|
return result
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def tokenmax_with_stats(text: str, tokenizer, force_cjk: bool = False) -> tuple[str, dict]:
|
||||||
|
"""Like tokenmax() but also returns substitution statistics.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
force_cjk: If True, always apply when CJK substitutions were made,
|
||||||
|
even if total token count increased. Prioritizes CJK adoption
|
||||||
|
over token savings.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(processed_text, stats_dict) where stats_dict contains:
|
||||||
|
- original_tokens: token count before processing
|
||||||
|
- result_tokens: token count after processing
|
||||||
|
- saved: tokens saved (negative = token increase; check forced_cjk)
|
||||||
|
- applied: whether the processed version was used
|
||||||
|
- forced_cjk: True when force_cjk override caused acceptance despite no savings
|
||||||
|
- filler_drops, cjk, symbol, abbrev: substitution counts by category
|
||||||
|
- total_subs: total substitutions applied
|
||||||
|
"""
|
||||||
|
if not text or not text.strip():
|
||||||
|
return text, {'original_tokens': 0, 'result_tokens': 0, 'saved': 0,
|
||||||
|
'applied': False, 'forced_cjk': False, 'total_subs': 0}
|
||||||
|
|
||||||
|
original_tokens = _ntok(text, tokenizer)
|
||||||
|
result, stats = _apply_subs(text)
|
||||||
|
result_tokens = _ntok(result, tokenizer)
|
||||||
|
saved = original_tokens - result_tokens
|
||||||
|
|
||||||
|
stats['original_tokens'] = original_tokens
|
||||||
|
stats['result_tokens'] = result_tokens
|
||||||
|
stats['saved'] = saved
|
||||||
|
stats['forced_cjk'] = False
|
||||||
|
|
||||||
|
if saved > 0:
|
||||||
|
stats['applied'] = True
|
||||||
|
return result, stats
|
||||||
|
if force_cjk and stats.get('cjk', 0) > 0:
|
||||||
|
stats['applied'] = True
|
||||||
|
stats['forced_cjk'] = True
|
||||||
|
return result, stats
|
||||||
|
stats['applied'] = False
|
||||||
|
return text, stats
|
||||||
|
|
||||||
|
|
||||||
|
# ── CLI: batch process a JSONL file ────────────────────────────────────
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import json, sys, argparse
|
||||||
|
from transformers import AutoTokenizer
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description='Token-max post-processor for think blocks')
|
||||||
|
parser.add_argument('--input', required=True, help='Input JSONL (messages format)')
|
||||||
|
parser.add_argument('--output', help='Output JSONL (default: dry run, stats only)')
|
||||||
|
parser.add_argument('--tokenizer', default='ZelligeAI/tessera-compressor',
|
||||||
|
help='HF repo id or local path of the tokenizer to count savings under')
|
||||||
|
parser.add_argument('--force-cjk', action='store_true',
|
||||||
|
help='Force CJK substitutions even if total tokens increase. '
|
||||||
|
'Prioritizes CJK adoption over token savings.')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
tok = AutoTokenizer.from_pretrained(args.tokenizer, trust_remote_code=True)
|
||||||
|
|
||||||
|
total_before = total_after = applied = skipped = forced = 0
|
||||||
|
|
||||||
|
out_lines = []
|
||||||
|
with open(args.input) as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
rec = json.loads(line)
|
||||||
|
for m in rec.get('messages', rec.get('conversations', [])):
|
||||||
|
role = m.get('role', m.get('from', ''))
|
||||||
|
if role not in ('assistant', 'gpt'):
|
||||||
|
continue
|
||||||
|
content_key = 'content' if 'content' in m else 'value'
|
||||||
|
c = m.get(content_key, '') or ''
|
||||||
|
if '<think>' not in c or '</think>' not in c:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Extract think content, preserving prefix before <think> and suffix after </think>
|
||||||
|
think_start = c.index('<think>') + len('<think>')
|
||||||
|
think_end = c.index('</think>')
|
||||||
|
prefix = c[:think_start - len('<think>')]
|
||||||
|
think = c[think_start:think_end]
|
||||||
|
suffix = c[think_end + len('</think>'):]
|
||||||
|
|
||||||
|
maxed, stats = tokenmax_with_stats(think, tok, force_cjk=args.force_cjk)
|
||||||
|
|
||||||
|
total_before += stats['original_tokens']
|
||||||
|
if stats['applied']:
|
||||||
|
applied += 1
|
||||||
|
total_after += stats['result_tokens']
|
||||||
|
m[content_key] = f'{prefix}<think>{maxed}</think>{suffix}'
|
||||||
|
if stats.get('forced_cjk'):
|
||||||
|
forced += 1
|
||||||
|
else:
|
||||||
|
skipped += 1
|
||||||
|
total_after += stats['original_tokens']
|
||||||
|
|
||||||
|
out_lines.append(json.dumps(rec, ensure_ascii=False))
|
||||||
|
|
||||||
|
if args.output:
|
||||||
|
with open(args.output, 'w') as f:
|
||||||
|
for line in out_lines:
|
||||||
|
f.write(line + '\n')
|
||||||
|
|
||||||
|
total = applied + skipped
|
||||||
|
saved = total_before - total_after
|
||||||
|
if total > 0:
|
||||||
|
print(f'Processed {total} think blocks')
|
||||||
|
print(f' Applied: {applied} ({100*applied/total:.0f}%)')
|
||||||
|
if forced:
|
||||||
|
print(f' Forced CJK: {forced} (applied despite no token savings)')
|
||||||
|
print(f' Skipped (no savings): {skipped}')
|
||||||
|
pct = f'{100*saved/total_before:.1f}' if total_before > 0 else '0.0'
|
||||||
|
print(f' Tokens: {total_before} → {total_after} = {saved:+d} ({pct}%)')
|
||||||
|
else:
|
||||||
|
print(f'No think blocks found in {args.input}')
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:ea43b288542655d72d632195ab9b58ca2cd9532c292bf6667827ce899ad196bc
|
||||||
|
size 11422082
|
||||||
217
tokenizer_config.json
Normal file
217
tokenizer_config.json
Normal file
@@ -0,0 +1,217 @@
|
|||||||
|
{
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"is_local": false,
|
||||||
|
"model_max_length": 32768,
|
||||||
|
"pad_token": "<|PAD_TOKEN|>",
|
||||||
|
"padding_side": "left",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<tool_call>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "</tool_call>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151659": {
|
||||||
|
"content": "<|fim_prefix|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151660": {
|
||||||
|
"content": "<|fim_middle|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151661": {
|
||||||
|
"content": "<|fim_suffix|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151662": {
|
||||||
|
"content": "<|fim_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151663": {
|
||||||
|
"content": "<|repo_name|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151664": {
|
||||||
|
"content": "<|file_sep|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151665": {
|
||||||
|
"content": "<|PAD_TOKEN|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %} {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user