183 lines
7.1 KiB
Markdown
183 lines
7.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
base_model:
|
||
- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- text-generation-inference
|
||
model-index:
|
||
- name: QwQ-R1-Distill-7B-CoT
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: IFEval (0-Shot)
|
||
type: wis-k/instruction-following-eval
|
||
split: train
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: inst_level_strict_acc and prompt_level_strict_acc
|
||
value: 35.0
|
||
name: averaged accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FQwQ-R1-Distill-7B-CoT
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: BBH (3-Shot)
|
||
type: SaylorTwift/bbh
|
||
split: test
|
||
args:
|
||
num_few_shot: 3
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 20.95
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FQwQ-R1-Distill-7B-CoT
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MATH Lvl 5 (4-Shot)
|
||
type: lighteval/MATH-Hard
|
||
split: test
|
||
args:
|
||
num_few_shot: 4
|
||
metrics:
|
||
- type: exact_match
|
||
value: 27.19
|
||
name: exact match
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FQwQ-R1-Distill-7B-CoT
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: GPQA (0-shot)
|
||
type: Idavidrein/gpqa
|
||
split: train
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 5.82
|
||
name: acc_norm
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FQwQ-R1-Distill-7B-CoT
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MuSR (0-shot)
|
||
type: TAUR-Lab/MuSR
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 4.5
|
||
name: acc_norm
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FQwQ-R1-Distill-7B-CoT
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MMLU-PRO (5-shot)
|
||
type: TIGER-Lab/MMLU-Pro
|
||
config: main
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 20.05
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FQwQ-R1-Distill-7B-CoT
|
||
name: Open LLM Leaderboard
|
||
---
|
||
# **QwQ-R1-Distill-7B-CoT**
|
||
|
||
QwQ-R1-Distill-7B-CoT is based on the *Qwen [ KT ] model*, which was distilled by DeepSeek-R1-Distill-Qwen-7B. It has been fine-tuned on the long chain-of-thought reasoning model and specialized datasets, focusing on chain-of-thought (CoT) reasoning for problem-solving. This model is optimized for tasks requiring logical reasoning, detailed explanations, and multi-step problem-solving, making it ideal for applications such as instruction-following, text generation, and complex reasoning tasks.
|
||
|
||
# **Quickstart with Transformers**
|
||
|
||
Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_name = "prithivMLmods/QwQ-R1-Distill-7B-CoT"
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_name,
|
||
torch_dtype="auto",
|
||
device_map="auto"
|
||
)
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
|
||
prompt = "Give me a short introduction to large language model."
|
||
messages = [
|
||
{"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
|
||
{"role": "user", "content": prompt}
|
||
]
|
||
text = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=False,
|
||
add_generation_prompt=True
|
||
)
|
||
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
||
|
||
generated_ids = model.generate(
|
||
**model_inputs,
|
||
max_new_tokens=512
|
||
)
|
||
generated_ids = [
|
||
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
|
||
]
|
||
|
||
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
||
```
|
||
### **Intended Use:**
|
||
1. **Instruction-Following:** The model excels in understanding and executing detailed instructions, making it ideal for automation systems, virtual assistants, and educational tools.
|
||
2. **Text Generation:** It can produce coherent, logically structured, and contextually relevant text for use in content creation, summarization, and report writing.
|
||
3. **Complex Reasoning Tasks:** With its fine-tuning for chain-of-thought reasoning, the model is well-suited for multi-step problem-solving, logical deduction, and question-answering tasks.
|
||
4. **Research and Development:** It can support researchers and developers in exploring advancements in logical reasoning and fine-tuning methodologies.
|
||
5. **Educational Applications:** The model can assist in teaching logical reasoning and problem-solving by generating step-by-step solutions.
|
||
|
||
### **Limitations:**
|
||
1. **Domain-Specific Knowledge:** While fine-tuned on reasoning datasets, the model may lack deep expertise in highly specialized or technical domains.
|
||
2. **Hallucination:** Like many large language models, it can generate incorrect or fabricated information, especially when reasoning beyond its training data.
|
||
3. **Bias in Training Data:** The model's outputs may reflect biases present in the datasets it was fine-tuned on, which could limit its objectivity in certain contexts.
|
||
4. **Performance on Non-Reasoning Tasks:** The model is optimized for chain-of-thought reasoning and may underperform on tasks that require simpler, less structured responses.
|
||
5. **Resource-Intensive:** Running the model efficiently requires significant computational resources, which may limit accessibility for smaller-scale deployments.
|
||
6. **Dependence on Input Quality:** The model’s performance heavily depends on the clarity and quality of the input provided. Ambiguous or poorly structured prompts may yield suboptimal results.
|
||
|
||
|
||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/prithivMLmods__QwQ-R1-Distill-7B-CoT-details)!
|
||
Summarized results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/contents/viewer/default/train?q=prithivMLmods%2FQwQ-R1-Distill-7B-CoT&sort[column]=Average%20%E2%AC%86%EF%B8%8F&sort[direction]=desc)!
|
||
|
||
| Metric |Value (%)|
|
||
|-------------------|--------:|
|
||
|**Average** | 18.92|
|
||
|IFEval (0-Shot) | 35.00|
|
||
|BBH (3-Shot) | 20.95|
|
||
|MATH Lvl 5 (4-Shot)| 27.19|
|
||
|GPQA (0-shot) | 5.82|
|
||
|MuSR (0-shot) | 4.50|
|
||
|MMLU-PRO (5-shot) | 20.05|
|
||
|