QwQ-32B-Preview-AWQ/README.md

---
license: apache-2.0
base_model:
- Qwen/QwQ-32B-Preview
language:
- en
pipeline_tag: text-generation
library_name: transformers
---
# QwQ-32B-Preview AWQ 4-Bit Quantized Version

## Introduction

This repository provides the **AWQ 4-bit quantized** version of the **QwQ-32B-Preview** model, originally developed by the Qwen Team. The quantized model significantly reduces memory usage and computational requirements, making it suitable for deployment on hardware with limited resources.

**Note**: This quantized model requires approximately **20 GB of VRAM** to run effectively.

**QwQ-32B-Preview** is an experimental research model aimed at advancing AI reasoning capabilities, particularly in mathematics and coding tasks. While it shows promising analytical abilities, it has several important limitations:

- **Language Mixing and Code Switching**: The model may unexpectedly switch between languages or mix them, affecting the clarity of responses.
- **Recursive Reasoning Loops**: There's a possibility of the model entering circular reasoning patterns, leading to lengthy responses without conclusive answers.
- **Safety and Ethical Considerations**: Enhanced safety measures are needed to ensure reliable and secure performance. Users should exercise caution when deploying the model.
- **Performance Limitations**: While excelling in math and coding, the model may underperform in areas like common sense reasoning and nuanced language understanding.

---

## Requirements

Ensure you are using the latest version of Hugging Face Transformers, as the code for Qwen2.5 is integrated there. Using a version earlier than **4.37.0** may result in the following error:

```plaintext
KeyError: 'qwen2'
```

---

## Quickstart

Here's how to load the tokenizer and model, and generate content using the quantized model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "KirillR/QwQ-32B-Preview-AWQ"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "How many 'r's are in 'strawberry'?"
messages = [
    {"role": "system", "content": "You are a helpful assistant developed by Alibaba. Please think step-by-step."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=1024
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(response)
```

---

## Original Model

For more details about the original QwQ-32B-Preview model, please refer to the following resource:

https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct-AWQ


---

## Citation

If you find the original model helpful, please consider citing the original authors:

```bibtext
@misc{qwq-32b-preview,
    title = {QwQ: Reflect Deeply on the Boundaries of the Unknown},
    url = {https://qwenlm.github.io/blog/qwq-32b-preview/},
    author = {Qwen Team},
    month = {November},
    year = {2024}
}

@article{qwen2,
      title={Qwen2 Technical Report}, 
      author={An Yang and Baosong Yang and others},
      journal={arXiv preprint arXiv:2407.10671},
      year={2024}
}
```