tessera-compressor/README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
language:
  - en
  - zh
pipeline_tag: text-generation
tags:
  - reasoning-compression
  - cjk
  - chain-of-thought
  - distillation
  - qwen2.5
---

![tessera-compressor](banner.png)

# tessera-compressor

A 1.5B model that compresses English reasoning text into a telegraphic CJK/symbol register under deterministic fidelity gates. It minted the training data for [Tessera-Preview-9B](https://huggingface.co/ZelligeAI/tessera-preview-9b) and replaces the frontier-model teacher that originally produced the register: English reasoning text becomes compressed-register training data at local-inference cost, with no API key and no external dependency. Validation covered code-centric reasoning (103 held-out mixed blocks); behavior on distant domains is unmeasured.

**Paper:** [Tessera-Preview-9B: Compressed Reasoning at 18x Fewer Tokens, and What It Costs](https://zellige.ai/research/compressed-cjk-reasoning) — section 3.1 covers this compressor's design and acceptance record.

Example (real training pair, 85 to 49 tokens):

```text
EN : So the classes are: - Integer (line 32) - Boolean (line 262) - BitString (line 341)
     - OctetString (line 693) ... Let me look at the base class to see if it defines __mul__:

CJK: Integer(line32),Boolean(line262),BitString(line341),OctetString(line693). 查基类是否定义__mul__:
```

## How it works

The compressor operates on passages, not whole blocks. A reasoning block is segmented (code fences stay atomic), sentences are grouped into step-sized passages, each passage is classified as fact-dense or narrative, and the model compresses it against the tail of the chain built so far. Every model output then passes a deterministic gate: the passage's novel numbers and identifiers must survive as substrings, the output must not blow up in length, and it must not exceed a rules-only compression of the same passage in token count. A passage that fails any check falls back to the rules-only version, so a bad generation costs savings rather than gated content. The gate is lexical, not semantic: it prevents the loss of numbers and identifiers, and a judged semantic-equivalence check backed it at acceptance (below), but it does not by itself guarantee semantic preservation on arbitrary input.

## Acceptance record

Measured on 103 held-out reasoning blocks the model never trained on, under criteria fixed before evaluation:

| Criterion | Result |
| --- | --- |
| Per-passage fidelity gate (numbers and identifiers survive) | 99.0% |
| Median per-passage compression ratio (output/input tokens) | 0.716 |
| CJK adoption | 98.9% of compressed passages |
| Judged semantic equivalence | 103/103 blocks (teacher references on the same blocks: 97.1%) |
| Degenerate outputs | 0 |
| Net corpus savings (after 24% rules-only fallback) | 30.4% |

On whole thinks in downstream production use (45,202 pairs), the compressed rendering costs a median 0.58x the tokens of its English source.

## Files

- Root: merged model, standard Hugging Face format (bf16). Base: Qwen2.5-Coder-1.5B-Instruct, LoRA r=16 merged in.
- `gguf/compressor-v31-q8_0.gguf`: llama.cpp quantization, validated behaviorally (scores 4/4 on the same acceptance suite). q4_k_m showed visible drift and is not published.
- `scripts/`: the complete usage harness. No tokens or keys required anywhere.

## Usage

Serve the model behind any OpenAI-compatible endpoint:

```bash
vllm serve ZelligeAI/tessera-compressor --port 8001
# or, CPU-friendly:
llama-server -m gguf/compressor-v31-q8_0.gguf --port 8001
```

Then run the harness:

```bash
cd scripts && pip install -r requirements.txt

# compress one reasoning block from a text file
python compress.py --in think.txt --endpoint http://localhost:8001/v1

# compress a corpus: {"id": ..., "text": ...} per JSONL line
python compress.py --in blocks.jsonl --out compressed.jsonl \
    --endpoint http://localhost:8001/v1
```

Output records carry the compressed text, source and output token counts, and per-block harness stats (model-accepted vs rules-fallback passage counts).

`scripts/` contents:

- `compress.py`: the driver. Segment, classify, compress per passage with chain context, gate, fall back on failure.
- `segmenting.py`: segmentation, passage grouping, fact extraction, classification, and the fidelity gate. Pure text processing.
- `tokenmax.py`: deterministic token-saving substitutions, used as the rules-only fallback and as a post-processor.

One note on token counting: the gate compares token counts under a tokenizer you choose (`--tokenizer`, default this repo). To reproduce the acceptance harness exactly, point it at the tokenizer of the model you are minting data for (the acceptance run used the Qwen3.5 target tokenizer).

Throughput on the acceptance hardware was 19.6K blocks/hour on one GPU, which makes minting compressed data cheap at any corpus size.

## License

Apache-2.0, same as the base model.