qwen2.5-coder-7b-pandas-dpo-aligned/README.md

library_name, tags

library_name	tags
transformers

qwen2.5-coder-7b-pandas-dpo-aligned

Introduction

Fine-tuned version of Qwen's Code-Specific large language model, Qwen2.5-coder-7b. Qwen-2.5 Coder has six mainstream model sizes: 0.5, 1.5, 3, 7, 14 and 32 billion parameter models to meet developer needs. The 7B model version provides a great balance between model representationl capacity and runtime requirements. This 7 Billion parameter model was fine-tuned using offline DPO and a preference dataset for Python data analysis.

Available in GGUF format.

• Q4_K_M recommended for systems with at least 8GB of RAM. Model file size: 4.68GB

Context Window: up to 128K tokens

Model Description

• License: Apache 2.0
• Finetuned from model: Qwen/Qwen2.5-Coder-7B-Instruct

Uses

Provides a good foundation for real-world applications such as Coding Agents. It has enhanced coding capabilities and maintains strengths in mathematics and general competencies.

Evaluation

Compute Infrastructure

The memory you need depends heavily on precision/quantization and context length. Always consider headroom for runtime and cache! KV/Key-Value cache (scales with context length and batch size).

Considering standard Memory (bytes)≈params×bytes per parameter rule of thumb:

Example 1 (CPU): Running on CPU (llama.cpp / GGUF):

A 7B model at 4‑bit needs about 3.5 – 4GB (7×10^9x0.5) for weights. With runtime overhead, KV cache, and your app, a practical minimum is:

• 16 GB RAM (barely)
• 32 GB RAM recommended for smoother operation

Example 2 (GPU): For local GPU inference with decent context and speed:

• Aim for 10–12 GB VRAM (7×10^9×0.5) for a 4‑bit Qwen2.5‑Coder‑7B.
• Aim for 16 GB VRAM (7×10^9×2) if you want FP16 or very long contexts.

Hardware

No GPU offload:

Absolute Minimum: 4 vCPUs (4 cores / threads) to run 7B 4‑bit model, but generation will likely feel sluggish, especially with longer responses.

Realistic Minimum: For heavier coding uses with multiple users and long context: 8–12 vCPUs, allows:

• 8–10 threads for the model.
• Remaining cores for the app and OS.

With GPU offload performing all the matrix multiplication 4 vCPUs are sufficient.

More Information [optional]

[More Information Needed]