40 lines
1.5 KiB
Markdown
40 lines
1.5 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: mit
|
|
base_model:
|
|
- deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
|
|
base_model_relation: quantized
|
|
---
|
|
|
|
# GPTQ 4bit quantized version of [DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B)
|
|
|
|
|
|
## Model Details
|
|
|
|
See details on the official page of the model: [DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B)
|
|
|
|
Quantized using [GPTQModel](https://github.com/ModelCloud/GPTQModel) using [wikitext2 dataset](https://github.com/ModelCloud/GPTQModel/blob/main/examples/quantization/basic_usage_wikitext2.py) with `nsamples=256` and `seqlen=1024`. Quantization config:
|
|
```
|
|
bits=4,
|
|
group_size=128,
|
|
desc_act=False,
|
|
damp_percent=0.01,
|
|
```
|
|
Minimum VRAM required: ~11GB
|
|
|
|
## How to use
|
|
Using `transformers` library with integrated GPTQ support:
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
|
|
|
|
model_name = "avoroshilov/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
quantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda')
|
|
|
|
chat = [{"role": "user", "content": "Why is grass green?"},]
|
|
question_tokens = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_tensors="pt").to(quantized_model.device)
|
|
answer_tokens = quantized_model.generate(question_tokens, generation_config=GenerationConfig(max_length=2048, ))[0]
|
|
|
|
print(tokenizer.decode(answer_tokens))
|
|
``` |