Qwen2.5-Coder-7B-Instruct-AWQ/README.md

license, license_link, language, base_model, pipeline_tag, library_name, tags

license	license_link	language	base_model	pipeline_tag	library_name	tags
apache-2.0	https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/blob/main/LICENSE	en	Qwen/Qwen2.5-Coder-7B-Instruct	text-generation	transformers	code codeqwen chat qwen qwen-coder

Qwen2.5-Coder-7B-Instruct-AWQ

Introduction

Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). As of now, Qwen2.5-Coder has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters, to meet the needs of different developers. Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:

• Significantly improvements in code generation, code reasoning and code fixing. Base on the strong Qwen2.5, we scale up the training tokens into 5.5 trillion including source code, text-code grounding, Synthetic data, etc. Qwen2.5-Coder-32B has become the current state-of-the-art open-source codeLLM, with its coding abilities matching those of GPT-4o.
• A more comprehensive foundation for real-world applications such as Code Agents. Not only enhancing coding capabilities but also maintaining its strengths in mathematics and general competencies.
• Long-context Support up to 128K tokens.

This repo contains the AWQ-quantized 4-bit instruction-tuned 7B Qwen2.5-Coder model, which has the following features:

• Type: Causal Language Models
• Training Stage: Pretraining & Post-training
• Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
• Number of Parameters: 7.61B
• Number of Paramaters (Non-Embedding): 6.53B
• Number of Layers: 28
• Number of Attention Heads (GQA): 28 for Q and 4 for KV
• Context Length: Full 131,072 tokens
  • Please refer to this section for detailed instructions on how to deploy Qwen2.5 for handling long texts.
• Quantization: AWQ 4-bit

For more details, please refer to our blog, GitHub, Documentation, Arxiv.

Requirements

The code of Qwen2.5-Coder has been in the latest Hugging face transformers and we advise you to use the latest version of transformers.

With transformers<4.37.0, you will encounter the following error:

KeyError: 'qwen2'

This model depends on gptqmodel. Please pin the version to 7.3.1 (newer versions are currently incompatible):

pip install gptqmodel==7.3.1

Also check out our AWQ documentation for more usage guide.