76 lines
2.7 KiB
Markdown
76 lines
2.7 KiB
Markdown
|
---
|
||
|
|
base_model: unsloth/Qwen2.5-7B-Instruct-bnb-4bit
|
||
|
|
tags:
|
||
|
|
- text-generation-inference
|
||
|
|
- transformers
|
||
|
|
- unsloth
|
||
|
|
- qwen2
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
---
|
||
|
|
|
||
|
|
# Uploaded finetuned model
|
||
|
|
|
||
|
|
- **Developed by:** bqbbao6
|
||
|
|
- **License:** apache-2.0
|
||
|
|
- **Finetuned from model :** unsloth/Qwen2.5-7B-Instruct-bnb-4bit
|
||
|
|
|
||
|
|
This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
|
||
|
|
|
||
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# Vietnamese Legal Qwen2.5 (7B Instruct - 4-bit Quantized)
|
||
|
|
|
||
|
|
**Model:** [bqbbao6/Qwen2.5-7B-legal-vn](https://huggingface.co/bqbbao6/Qwen2.5-7B-legal-vn)
|
||
|
|
|
||
|
|
**Version 3B:** [bqbbao6/Qwen2.5-3B-legal-vn](https://huggingface.co/bqbbao6/Qwen2.5-3B-legal-vn)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Model Description
|
||
|
|
|
||
|
|
This model is a fine-tuned version of **Qwen2.5-7B-Instruct**, optimized using Unsloth and quantized to **4-bit (bitsandbytes)** .
|
||
|
|
|
||
|
|
The primary fine-tuning objective focuses on **Vietnamese Legal Domain Mastery** and **Context-Based Question Answering (RAG)**. Unlike general-purpose LLMs that reply strictly from pre-trained parametric memory, this model is specifically aligned to prioritize and reason directly over the provided context (legal articles, decrees, and circulars), drastically reducing hallucination and ensuring high-fidelity legal consulting.
|
||
|
|
|
||
|
|
### Key Enhancements:
|
||
|
|
* **Strict Context Adherence:** Tuned to extract facts and formulate arguments heavily based on the input context—making it perfect for Retrieval-Augmented Generation (RAG) pipelines.
|
||
|
|
* **Legal Formalism:** Adopts the authoritative, formal, and precise tone required in the Vietnamese administrative and legal sectors.
|
||
|
|
* **Hardware Efficiency:** Operates smoothly within a ~6GB VRAM footprint during inference, leaving ample head-room for long contexts and tool-calling structures.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
| Property | Value |
|
||
|
|
|---|---|
|
||
|
|
| **Base Model** | Qwen/Qwen2.5-7B-Instruct |
|
||
|
|
| **Parameters** | 7 Billion |
|
||
|
|
| **Quantization** | 4-bit (bitsandbytes / bnb-4bit) |
|
||
|
|
| **Fine-tuning Method** | QLoRA (Rank 16, Alpha 32) |
|
||
|
|
| **Primary Task** | Context-Driven Legal Q&A / Legal Agent |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Deployment & Inference (vLLM)
|
||
|
|
|
||
|
|
To host this model using **vLLM** on standard cloud environments.
|
||
|
|
|
||
|
|
### Start the vLLM Server:
|
||
|
|
```bash
|
||
|
|
!python -m vllm.entrypoints.openai.api_server \
|
||
|
|
--model unsloth/Qwen2.5-7B-Instruct-bnb-4bit \
|
||
|
|
--max-model-len 2048 \ # Can be changed
|
||
|
|
--dtype float16 \
|
||
|
|
--api-key 'your-api-key-here' \
|
||
|
|
--max-num-seqs 16 \
|
||
|
|
--trust-remote-code \
|
||
|
|
--gpu-memory-utilization 0.85 \ # Can be changed
|
||
|
|
--enforce-eager \
|
||
|
|
--enable-auto-tool-choice \
|
||
|
|
--tool-call-parser hermes \
|
||
|
|
--port 8000 &
|