88 lines
2.7 KiB
Markdown
88 lines
2.7 KiB
Markdown
---
|
||
library_name: transformers
|
||
tags: []
|
||
---
|
||
|
||
# qwen2.5-coder-7b-pandas-dpo-aligned
|
||
|
||
<!-- Provide a quick summary of what the model is/does. -->
|
||
|
||
## Introduction
|
||
|
||
Fine-tuned version of Qwen's Code-Specific large language model, Qwen2.5-coder-7b. Qwen-2.5 Coder has six mainstream model sizes: 0.5, 1.5, 3, 7, 14 and 32 billion
|
||
parameter models to meet developer needs. The 7B model version provides a great balance between model representationl capacity and runtime requirements. This 7 Billion parameter
|
||
model was fine-tuned using offline DPO and a preference dataset for Python data analysis.
|
||
|
||
Available in GGUF format.
|
||
- Q4_K_M recommended for systems with at least 8GB of RAM. Model file size: 4.68GB
|
||
|
||
**Context Window:** up to 128K tokens
|
||
|
||
### Model Description
|
||
|
||
<!-- Provide a longer summary of what this model is. -->
|
||
|
||
|
||
|
||
|
||
- **License:** Apache 2.0
|
||
- **Finetuned from model:** [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)
|
||
|
||
|
||
|
||
|
||
## Uses
|
||
|
||
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
||
|
||
Provides a good foundation for real-world applications such as Coding Agents. It has enhanced coding capabilities and
|
||
maintains strengths in mathematics and general competencies.
|
||
|
||
|
||
|
||
## Evaluation
|
||
|
||
<!-- This section describes the evaluation protocols and provides the results. -->
|
||
|
||
|
||
|
||
|
||
### Compute Infrastructure
|
||
|
||
The memory you need depends heavily on precision/quantization and context length. *Always consider headroom for runtime and cache!*
|
||
KV/Key-Value cache (scales with context length and batch size).
|
||
|
||
Considering standard Memory (bytes)≈params×bytes per parameter rule of thumb:
|
||
|
||
**Example 1 (CPU):** Running on CPU (llama.cpp / GGUF):
|
||
|
||
A 7B model at 4‑bit needs about 3.5 – 4GB (7×10^9x0.5) for weights. With runtime overhead, KV cache, and your app, a practical minimum is:
|
||
|
||
- 16 GB RAM (barely)
|
||
- 32 GB RAM recommended for smoother operation
|
||
|
||
**Example 2 (GPU):** For local GPU inference with decent context and speed:
|
||
|
||
- Aim for 10–12 GB VRAM (7×10^9×0.5) for a 4‑bit Qwen2.5‑Coder‑7B.
|
||
- Aim for 16 GB VRAM (7×10^9×2) if you want FP16 or very long contexts.
|
||
|
||
|
||
#### Hardware
|
||
|
||
No GPU offload:
|
||
|
||
**Absolute Minimum:** 4 vCPUs (4 cores / threads) to run 7B 4‑bit model, but generation will likely feel sluggish, especially with longer responses.
|
||
|
||
**Realistic Minimum:** For heavier coding uses with multiple users and long context: 8–12 vCPUs, allows:
|
||
|
||
- 8–10 threads for the model.
|
||
- Remaining cores for the app and OS.
|
||
|
||
With GPU offload performing all the matrix multiplication 4 vCPUs are sufficient.
|
||
|
||
|
||
## More Information [optional]
|
||
|
||
[More Information Needed]
|
||
|