70 lines
2.0 KiB
Markdown
70 lines
2.0 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
license: apache-2.0
|
||
|
|
license_link: https://huggingface.co/Qwen/Qwen3-32B/blob/main/LICENSE
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
base_model: Qwen/Qwen3-32B
|
||
|
|
---
|
||
|
|
|
||
|
|
# W8A8 INT8 Quantization of Qwen3-32B
|
||
|
|
|
||
|
|
I made this for running on vLLM with Ampere GPU.
|
||
|
|
|
||
|
|
On 2xRTX3090, you may set context length to upto 16384 (or 24576 if you use `--kv-cache-dtype fp8`).
|
||
|
|
|
||
|
|
|
||
|
|
## Quantization method
|
||
|
|
|
||
|
|
Quantized using
|
||
|
|
|
||
|
|
- Tool: [llmcompressor 0.5.2.dev22 (db959a3)](https://github.com/vllm-project/llm-compressor/commit/db959a3ec0c4a96b06698bc10e2d81016f5e8751).
|
||
|
|
- System: 4x3090, DDR4 128GB + swap 32GB
|
||
|
|
- Time taken: 6 hours (wall time)
|
||
|
|
|
||
|
|
|
||
|
|
```py
|
||
|
|
## modified based on the code from https://huggingface.co/nytopop/Qwen3-14B.w8a8
|
||
|
|
|
||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
|
from datasets import load_dataset
|
||
|
|
from llmcompressor import oneshot
|
||
|
|
from llmcompressor.modifiers.quantization import GPTQModifier
|
||
|
|
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
|
||
|
|
from llmcompressor.transformers.compression.helpers import calculate_offload_device_map
|
||
|
|
|
||
|
|
model_id = "Qwen/Qwen3-32B"
|
||
|
|
model_out = "Qwen3-32B.w8a8"
|
||
|
|
|
||
|
|
num_samples = 256
|
||
|
|
max_seq_len = 4096
|
||
|
|
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
|
|
||
|
|
def preprocess_fn(example):
|
||
|
|
return {"text": tokenizer.apply_chat_template(example["messages"], add_generation_prompt=False, tokenize=False)}
|
||
|
|
|
||
|
|
ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")
|
||
|
|
ds = ds.shuffle().select(range(num_samples))
|
||
|
|
ds = ds.map(preprocess_fn)
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
model_id,
|
||
|
|
device_map="auto",
|
||
|
|
torch_dtype="bfloat16",
|
||
|
|
max_memory={0: "10GiB", 1:"10GiB", 2:"10GiB", 3:"10GiB", "cpu":"96GiB"},
|
||
|
|
)
|
||
|
|
|
||
|
|
recipe = [
|
||
|
|
SmoothQuantModifier(smoothing_strength=0.7),
|
||
|
|
GPTQModifier(sequential=True,targets="Linear",scheme="W8A8",ignore=["lm_head"],dampening_frac=0.01),
|
||
|
|
]
|
||
|
|
|
||
|
|
oneshot(
|
||
|
|
model=model,
|
||
|
|
dataset=ds,
|
||
|
|
recipe=recipe,
|
||
|
|
max_seq_length=max_seq_len,
|
||
|
|
num_calibration_samples=num_samples,
|
||
|
|
output_dir=model_out,
|
||
|
|
)
|
||
|
|
```
|