32 lines
877 B
Markdown
32 lines
877 B
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen3-4B-Instruct-2507
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen3-4B-Instruct-2507-GPTQ-INT4
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
This model is a int4 model with group_size 128 of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) generated by [vastai modelzoo](https://github.com/Vastai/VastModelZOO). Please follow the license of the original model.
|
||
|
|
|
||
|
|
## vllm Inference
|
||
|
|
|
||
|
|
- vllm >= v0.11.0
|
||
|
|
- VVI >= 26.02
|
||
|
|
|
||
|
|
```bash
|
||
|
|
vllm serve vastai-ais/Qwen3-4B-Instruct-2507-GPTQ-INT4 --served_model_name qwen
|
||
|
|
```
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d ' {
|
||
|
|
"model": "qwen",
|
||
|
|
"messages": [
|
||
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
||
|
|
{"role": "user", "content": "Who are you?"}
|
||
|
|
],
|
||
|
|
"temperature": 1,
|
||
|
|
"max_tokens": 512
|
||
|
|
} '
|
||
|
|
```
|