151 lines
5.5 KiB
Markdown
151 lines
5.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
language:
|
|
- en
|
|
base_model: Qwen/Qwen2.5-3B-Instruct
|
|
tags:
|
|
- dystrio
|
|
- sculpt
|
|
- pruned
|
|
- compressed
|
|
- efficient
|
|
- dense
|
|
- runtime-agnostic
|
|
- no-custom-kernels
|
|
- hf-drop-in
|
|
- drop-in-replacement
|
|
- smaller
|
|
- faster
|
|
- qwen
|
|
datasets:
|
|
- wikitext
|
|
model-index:
|
|
- name: Dystrio Sculpt (Qwen2.5-3B-Instruct Default)
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
dataset:
|
|
name: WikiText-103 (validation)
|
|
type: wikitext
|
|
metrics:
|
|
- name: perplexity
|
|
type: perplexity
|
|
value: 15.5137
|
|
- name: ppl_ratio
|
|
type: ppl_ratio
|
|
value: 1.1079
|
|
---
|
|
|
|
# dystrio/Qwen2.5-3B-Instruct-sculpt-default
|
|
|
|
> **9% smaller, +6% faster prefill, drop-in replacement. No custom kernels. No runtime changes.**
|
|
|
|
Dystrio Sculpt structurally compresses transformer models, producing dense models that load with standard `transformers` — no custom code, no new ops, no deployment friction.
|
|
|
|
This is the **Default** tier of [Qwen2.5 3B Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct).
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("dystrio/Qwen2.5-3B-Instruct-sculpt-default", torch_dtype="bfloat16", device_map="auto")
|
|
tokenizer = AutoTokenizer.from_pretrained("dystrio/Qwen2.5-3B-Instruct-sculpt-default")
|
|
|
|
inputs = tokenizer("The future of AI inference is", return_tensors="pt").to(model.device)
|
|
outputs = model.generate(**inputs, max_new_tokens=100)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Benchmark Results
|
|
|
|
All tiers compiled from [Qwen2.5 3B Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) on A100 80GB, bf16:
|
|
|
|
| Model | PPL | PPL Ratio | Weights (GB) | Chat Prefill TPS | RAG TTFT p95 (ms) | Decode TPS |
|
|
|-------|-----|-----------|-------------|------------------|-------------------|------------|
|
|
| **Baseline** | 14.0033 | 1.0 | 5.748009 | 22079.6 | 75.483 | 59.4 |
|
|
| **sculpt-default** | 15.5137 | 1.1079 | 5.220665 | 23360.8 | 73.544 | 59.0 |
|
|
| **sculpt-production** | 18.9373 | 1.3523 | 4.956993 | 24367.4 | 69.025 | 59.0 |
|
|
| **sculpt-throughput** | 22.8847 | 1.6342 | 4.640587 | 25556.6 | 68.084 | 59.0 |
|
|
| **sculpt-experimental** | 31.3266 | 2.2371 | 4.165977 | 26731.9 | 66.499 | 59.5 |
|
|
|
|
### Key Metrics (this model)
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| **Weights memory** | 5.220665 GB (9% smaller) |
|
|
| **PPL ratio** | 1.1079 |
|
|
| **Chat prefill TPS** | 23360.8 (+6%) |
|
|
| **RAG TTFT p95** | 73.544 ms (-3%) |
|
|
| **Decode TPS** | 59.0 (flat) |
|
|
| **Parameters** | 2.80B |
|
|
|
|
## All Sculpt Tiers
|
|
|
|
| Tier | HuggingFace | Size | PPL Ratio | Use Case |
|
|
|------|-------------|------|-----------|----------|
|
|
| default | [dystrio/Qwen2.5-3B-Instruct-sculpt-default](https://huggingface.co/dystrio/Qwen2.5-3B-Instruct-sculpt-default) 👈 **this model** | 5.220665 GB | 1.1079 | Zero-regret: quality preserved, smaller footprint |
|
|
| production | [dystrio/Qwen2.5-3B-Instruct-sculpt-production](https://huggingface.co/dystrio/Qwen2.5-3B-Instruct-sculpt-production) | 4.956993 GB | 1.3523 | Practical savings with modest quality tradeoff |
|
|
| throughput | [dystrio/Qwen2.5-3B-Instruct-sculpt-throughput](https://huggingface.co/dystrio/Qwen2.5-3B-Instruct-sculpt-throughput) | 4.640587 GB | 1.6342 | Maximum usable compression for speed/edge |
|
|
| experimental | [dystrio/Qwen2.5-3B-Instruct-sculpt-experimental](https://huggingface.co/dystrio/Qwen2.5-3B-Instruct-sculpt-experimental) | 4.165977 GB | 2.2371 | Boundary exploration, maximum structural compression |
|
|
|
|
## What is Dystrio Sculpt?
|
|
|
|
Dystrio Sculpt compiles transformer models into smaller, faster variants. Output models:
|
|
|
|
- Are **dense** (not sparse) — standard architecture, fewer parameters
|
|
- Load with **standard HuggingFace Transformers** — no custom code needed
|
|
- Require **no custom kernels** and **no runtime changes**
|
|
- Work as a one-step compile before deployment
|
|
- Stack with quantization (AWQ, GPTQ, GGUF) for compound savings
|
|
|
|
## Compatibility
|
|
|
|
- ✅ HuggingFace Transformers
|
|
- ✅ vLLM
|
|
- ✅ TGI (Text Generation Inference)
|
|
- ✅ llama.cpp / GGUF conversion
|
|
- ✅ AWQ / GPTQ quantization
|
|
- ✅ Any framework that loads standard safetensors
|
|
|
|
## Benchmark Environment
|
|
|
|
- **GPU**: NVIDIA A100-SXM4-80GB
|
|
- **dtype**: bf16
|
|
- **Torch**: 2.10.0+cu128
|
|
- **Transformers**: 5.3.0
|
|
- **Deterministic**: True
|
|
- Single-GPU, standard HuggingFace Transformers, no custom kernels.
|
|
|
|
## Metric Definitions
|
|
|
|
- **PPL ratio**: WikiText-103 perplexity relative to baseline. <1.0 = quality improved.
|
|
- **Prefill TPS**: Tokens per second during prompt encoding (higher = faster).
|
|
- **TTFT p95**: Time to first token at 95th percentile (lower = faster).
|
|
- **Decode TPS**: Tokens per second during generation (higher = faster).
|
|
- **Weights (GB)**: Model parameter memory (deterministic, runtime-independent).
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@misc{dystrio_sculpt_2026,
|
|
title={Dystrio Sculpt: Structural Compilation for Transformer LLMs},
|
|
author={Dystrio},
|
|
year={2026},
|
|
url={https://huggingface.co/dystrio}
|
|
}
|
|
```
|
|
|
|
## Downstream Benchmarks (lm-eval)
|
|
|
|
Evaluated with [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness) on A100-80GB, bf16, zero-shot.
|
|
|
|
| Benchmark | Baseline | This Model | Delta |
|
|
|-----------|:--------:|:----------:|:-----:|
|
|
| ARC-Challenge | 0.4573 | 0.4266 | -0.0307 |
|
|
| HellaSwag | 0.5635 | 0.5045 | -0.0590 |
|
|
| MMLU | 0.6545 | 0.5789 | -0.0756 |
|
|
| TruthfulQA MC2 | 0.5874 | 0.5145 | -0.0729 |
|