13 lines
469 B
Markdown
13 lines
469 B
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen3-4B-Instruct-2507
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
This is my (first) attempt at quantizing this Qwen3 model (Qwen/Qwen3-4B-Instruct-2507) using auto-round, like so:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
auto-round --model "Qwen/Qwen3-4B-Instruct-2507" --scheme "W4A16" --format "auto_gptq" --output_dir "./Quantized" --model_dtype fp16
|
||
|
|
```
|
||
|
|
The primary purpose of these quants are to make them run on consumer AMD GPUs, which they do work, in my case.
|