149 lines
5.2 KiB
Markdown
149 lines
5.2 KiB
Markdown
---
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
language:
|
|
- en
|
|
base_model: Qwen/Qwen2.5-7B-Instruct
|
|
tags:
|
|
- dystrio
|
|
- sculpt
|
|
- pruned
|
|
- compressed
|
|
- efficient
|
|
- dense
|
|
- runtime-agnostic
|
|
- no-custom-kernels
|
|
- hf-drop-in
|
|
- drop-in-replacement
|
|
- smaller
|
|
- faster
|
|
- qwen
|
|
datasets:
|
|
- wikitext
|
|
model-index:
|
|
- name: Dystrio Sculpt (Qwen2.5-7B-Instruct Default)
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
dataset:
|
|
name: WikiText-103 (validation)
|
|
type: wikitext
|
|
metrics:
|
|
- name: perplexity
|
|
type: perplexity
|
|
value: 12.334
|
|
- name: ppl_ratio
|
|
type: ppl_ratio
|
|
value: 0.9896
|
|
---
|
|
|
|
# dystrio/Qwen2.5-7B-Instruct-sculpt-default
|
|
|
|
> **9% smaller, quality improved (0.9896x PPL), drop-in replacement. No custom kernels. No runtime changes.**
|
|
|
|
Dystrio Sculpt structurally compresses transformer models, producing dense models that load with standard `transformers` — no custom code, no new ops, no deployment friction.
|
|
|
|
This is the **Default** tier of [Qwen 2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("dystrio/Qwen2.5-7B-Instruct-sculpt-default", torch_dtype="bfloat16", device_map="auto")
|
|
tokenizer = AutoTokenizer.from_pretrained("dystrio/Qwen2.5-7B-Instruct-sculpt-default")
|
|
|
|
inputs = tokenizer("The future of AI inference is", return_tensors="pt").to(model.device)
|
|
outputs = model.generate(**inputs, max_new_tokens=100)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Benchmark Results
|
|
|
|
All tiers compiled from [Qwen 2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) on A100 80GB, bf16:
|
|
|
|
| Model | PPL | PPL Ratio | Weights (GB) | Chat Prefill TPS | RAG TTFT p95 (ms) | Decode TPS |
|
|
|-------|-----|-----------|-------------|------------------|-------------------|------------|
|
|
| **Baseline** | 12.4633 | 1.0 | 14.185191 | 11510.6 | 117.869 | 71.1 |
|
|
| **sculpt-default** | 12.334 | 0.9896 | 12.964976 | 12352.7 | 110.714 | 72.7 |
|
|
| **sculpt-production** | 21.9239 | 1.7591 | 10.596324 | 14700.3 | 95.291 | 73.5 |
|
|
| **sculpt-throughput** | 23.2366 | 1.8644 | 9.950328 | 15386.6 | 91.914 | 73.3 |
|
|
|
|
### Key Metrics (this model)
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| **Weights memory** | 12.964976 GB (9% smaller) |
|
|
| **PPL ratio** | 0.9896 |
|
|
| **Chat prefill TPS** | 12352.7 (+7%) |
|
|
| **RAG TTFT p95** | 110.714 ms (-6%) |
|
|
| **Decode TPS** | 72.7 (flat) |
|
|
| **Parameters** | 6.96B |
|
|
|
|
## All Sculpt Tiers
|
|
|
|
| Tier | HuggingFace | Size | PPL Ratio | Use Case |
|
|
|------|-------------|------|-----------|----------|
|
|
| default | [dystrio/Qwen2.5-7B-Instruct-sculpt-default](https://huggingface.co/dystrio/Qwen2.5-7B-Instruct-sculpt-default) 👈 **this model** | 12.964976 GB | 0.9896 | Zero-regret: quality preserved, smaller footprint |
|
|
| production | [dystrio/Qwen2.5-7B-Instruct-sculpt-production](https://huggingface.co/dystrio/Qwen2.5-7B-Instruct-sculpt-production) | 10.596324 GB | 1.7591 | Practical savings with modest quality tradeoff |
|
|
| throughput | [dystrio/Qwen2.5-7B-Instruct-sculpt-throughput](https://huggingface.co/dystrio/Qwen2.5-7B-Instruct-sculpt-throughput) | 9.950328 GB | 1.8644 | Maximum usable compression for speed/edge |
|
|
|
|
## What is Dystrio Sculpt?
|
|
|
|
Dystrio Sculpt compiles transformer models into smaller, faster variants. Output models:
|
|
|
|
- Are **dense** (not sparse) — standard architecture, fewer parameters
|
|
- Load with **standard HuggingFace Transformers** — no custom code needed
|
|
- Require **no custom kernels** and **no runtime changes**
|
|
- Work as a one-step compile before deployment
|
|
- Stack with quantization (AWQ, GPTQ, GGUF) for compound savings
|
|
|
|
## Compatibility
|
|
|
|
- ✅ HuggingFace Transformers
|
|
- ✅ vLLM
|
|
- ✅ TGI (Text Generation Inference)
|
|
- ✅ llama.cpp / GGUF conversion
|
|
- ✅ AWQ / GPTQ quantization
|
|
- ✅ Any framework that loads standard safetensors
|
|
|
|
## Benchmark Environment
|
|
|
|
- **GPU**: NVIDIA A100-SXM4-80GB
|
|
- **dtype**: bf16
|
|
- **Torch**: 2.10.0+cu128
|
|
- **Transformers**: 5.3.0
|
|
- **Deterministic**: True
|
|
- Single-GPU, standard HuggingFace Transformers, no custom kernels.
|
|
|
|
## Metric Definitions
|
|
|
|
- **PPL ratio**: WikiText-103 perplexity relative to baseline. <1.0 = quality improved.
|
|
- **Prefill TPS**: Tokens per second during prompt encoding (higher = faster).
|
|
- **TTFT p95**: Time to first token at 95th percentile (lower = faster).
|
|
- **Decode TPS**: Tokens per second during generation (higher = faster).
|
|
- **Weights (GB)**: Model parameter memory (deterministic, runtime-independent).
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@misc{dystrio_sculpt_2026,
|
|
title={Dystrio Sculpt: Structural Compilation for Transformer LLMs},
|
|
author={Dystrio},
|
|
year={2026},
|
|
url={https://huggingface.co/dystrio}
|
|
}
|
|
```
|
|
|
|
## Downstream Benchmarks (lm-eval)
|
|
|
|
Evaluated with [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness) on A100-80GB, bf16, zero-shot.
|
|
|
|
| Benchmark | Baseline | This Model | Delta |
|
|
|-----------|:--------:|:----------:|:-----:|
|
|
| ARC-Challenge | 0.5282 | 0.4676 | -0.0606 |
|
|
| HellaSwag | 0.6204 | 0.5650 | -0.0554 |
|
|
| MMLU | 0.7176 | 0.6506 | -0.0670 |
|
|
| TruthfulQA MC2 | 0.6475 | 0.5457 | -0.1018 |
|