129 lines
4.8 KiB
Markdown
129 lines
4.8 KiB
Markdown
---
|
||
license: mit
|
||
train: false
|
||
inference: true
|
||
pipeline_tag: text-generation
|
||
base_model:
|
||
- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
|
||
---
|
||
This is a version of the <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B">DeepSeek-R1-Distill-Qwen-7B</a> model re-distilled for better performance.
|
||
|
||
## Performance
|
||
|
||
| Models | <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B">DeepSeek-R1-Distill-Qwen-7B</a> | <a href="https://huggingface.co/mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1">DeepSeek-R1-ReDistill-Qwen-7B-v1.1</a> |
|
||
|:-------------------:|:--------:|:----------------:|
|
||
| ARC (25-shot) | <b>55.03</b> | 52.3 |
|
||
| HellaSwag (10-shot)| 61.9 | <b>62.36</b> |
|
||
| MMLU (5-shot) | 56.75 | <b>59.53</b> |
|
||
| TruthfulQA-MC2 | 45.76 | <b>47.7</b> |
|
||
| Winogrande (5-shot)| 60.38 | <b>61.8</b> |
|
||
| GSM8K (5-shot) | 78.85 | <b>83.4</b> |
|
||
| Average | 59.78 | <b>61.18</b> |
|
||
|
||
| Models | <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B">DeepSeek-R1-Distill-Qwen-7B</a> | <a href="https://huggingface.co/mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1">DeepSeek-R1-ReDistill-Qwen-7B-v1.1</a> |
|
||
|:-------------------:|:--------:|:----------------:|
|
||
| GPQA (0-shot) | 30.9 | <b>34.99</b> |
|
||
| MMLU PRO (5-shot) | 28.83 | <b>31.02</b> |
|
||
| MUSR (0-shot) | 38.85 | <b>44.42</b> |
|
||
| BBH (3-shot) | 43.54 | <b>51.53</b> |
|
||
| IfEval (0-shot) - strict | <b>42.33</b> | 35.49 |
|
||
| IfEval (0-shot) - loose | 30.31 | <b>38.49</b> |
|
||
|
||
## Usage
|
||
```Python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
compute_dtype = torch.bfloat16
|
||
device = 'cuda'
|
||
model_id = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1"
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa", device_map=device)
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
||
prompt = "What is 1.5+102.2?"
|
||
chat = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt")
|
||
outputs = model.generate(chat.to(device), max_new_tokens=1024, do_sample=True)
|
||
print(tokenizer.decode(outputs[0]))
|
||
```
|
||
|
||
Output:
|
||
```
|
||
<｜begin▁of▁sentence｜><｜User｜>What is 1.5+102.2?<｜Assistant｜><think>
|
||
First, I need to add the whole number parts of the two numbers. The whole numbers are 1 and 102, which add up to 103.
|
||
|
||
Next, I add the decimal parts of the two numbers. The decimal parts are 0.5 and 0.2, which add up to 0.7.
|
||
|
||
Finally, I combine the whole number and decimal parts to get the total sum. Adding 103 and 0.7 gives me 103.7.
|
||
</think>
|
||
|
||
To add the numbers \(1.5\) and \(102.2\), follow these steps:
|
||
|
||
1. **Add the whole number parts:**
|
||
\[
|
||
1 + 102 = 103
|
||
\]
|
||
|
||
2. **Add the decimal parts:**
|
||
\[
|
||
0.5 + 0.2 = 0.7
|
||
\]
|
||
|
||
3. **Combine the results:**
|
||
\[
|
||
103 + 0.7 = 103.7
|
||
\]
|
||
|
||
**Final Answer:**
|
||
\[
|
||
\boxed{103.7}
|
||
\]<｜end▁of▁sentence｜>
|
||
```
|
||
|
||
## HQQ
|
||
Run ~3.5x faster with <a href="https://github.com/mobiusml/hqq/">HQQ</a>. First, install the dependencies:
|
||
```
|
||
pip install hqq
|
||
```
|
||
|
||
```Python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
from hqq.models.hf.base import AutoHQQHFModel
|
||
from hqq.core.quantize import *
|
||
|
||
#Params
|
||
device = 'cuda:0'
|
||
backend = "torchao_int4"
|
||
compute_dtype = torch.bfloat16 if backend=="torchao_int4" else torch.float16
|
||
model_id = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1"
|
||
|
||
#Load
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa")
|
||
|
||
#Quantize
|
||
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, axis=1)
|
||
AutoHQQHFModel.quantize_model(model, quant_config=quant_config, compute_dtype=compute_dtype, device=device)
|
||
|
||
#Optimize
|
||
from hqq.utils.patching import prepare_for_inference
|
||
prepare_for_inference(model, backend=backend, verbose=False)
|
||
|
||
############################################################
|
||
#Generate (streaming)
|
||
from hqq.utils.generation_hf import HFGenerator
|
||
gen = HFGenerator(model, tokenizer, max_new_tokens=4096, do_sample=True, compile='partial').warmup()
|
||
|
||
prompt = "If A equals B, and C equals B - A, what would be the value of C?"
|
||
out = gen.generate(prompt, print_tokens=True)
|
||
|
||
############################################################
|
||
# #Generate (simple)
|
||
# from hqq.utils.generation_hf import patch_model_for_compiled_runtime
|
||
# patch_model_for_compiled_runtime(model, tokenizer, warmup=True)
|
||
|
||
# prompt = "If A equals B, and C equals B - A, what would be the value of C?"
|
||
# chat = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt")
|
||
# outputs = model.generate(chat.to(device), max_new_tokens=8192, do_sample=True)
|
||
# print(tokenizer.decode(outputs[0]))
|
||
``` |