bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-19 12:33:15 +08:00
commit b8d2bdf73d
28 changed files with 312 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct-f16.gguf filter=lfs diff=lfs merge=lfs -text
Llama-Doctor-3.2-3B-Instruct.imatrix filter=lfs diff=lfs merge=lfs -text

3
Llama-Doctor-3.2-3B-Instruct-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:19ba7034fd29e9d2a3d1012779e865ca8b93e82872687de53765ab2d8bbc0289
size 1229033152

3
Llama-Doctor-3.2-3B-Instruct-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c487a0268a3dc2928273cadd2b40c8fb7bf3b8ed570e1b5ab22d0f086d50384
size 1599669952

3
Llama-Doctor-3.2-3B-Instruct-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:08f9cb6cd7741ad818d9e6c3217fb6490ee611190bce9db26652cc23ed3b7f88
size 1476789952

3
Llama-Doctor-3.2-3B-Instruct-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:29937bfa42bff2f52632ae8b454ebab539a811a30102f9e985e721887e8de1a8
size 1829111488

3
Llama-Doctor-3.2-3B-Instruct-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cbfed829f1ee787a9f2d6e12ea26aba5428b14ee4649a7706fa2b873288519e5
size 1363936960

3
Llama-Doctor-3.2-3B-Instruct-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:551111fba93e77856d9ef9ffa49dddaeaba21cb163461ab5da99b0e8e500fccb
size 1459359424

3
Llama-Doctor-3.2-3B-Instruct-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:68177da8e5346a509d42b44519d0b11e8e7f7439017516c0410d48976d4ae702
size 1815348928

3
Llama-Doctor-3.2-3B-Instruct-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:555c0627e118f0e87d8a7155817d3a5d652e781b732f11a36f0279557907f6dc
size 1687160512

3
Llama-Doctor-3.2-3B-Instruct-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c6f1152c730606a1b15ff588094874b9c781f42d14896435179548e10a9828c3
size 1542850240

3
Llama-Doctor-3.2-3B-Instruct-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b39946ba651edcc5c39bde9c187a89f04a3a4f6f21147344d8b605a156e9ea41
size 1910771392

3
Llama-Doctor-3.2-3B-Instruct-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a7a0b9828bf13e7080e63adc3e6990707158eadc8681a695008e5cc96de243ae
size 1921910464

3
Llama-Doctor-3.2-3B-Instruct-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ebf45a7a9d77865f26994c4786b513951fc8164af38cfc7f67c287d5689b7089
size 1917191872

3
Llama-Doctor-3.2-3B-Instruct-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d8a13c26b6800262d915c71e15e72e452f91e403859a3dad0efed8e430834466
size 1917191872

3
Llama-Doctor-3.2-3B-Instruct-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:16c6a32f0aa813edd1d98c9f2b01f648c4b6d0eab13f0f8afa2bc2047fecf118
size 1917191872

3
Llama-Doctor-3.2-3B-Instruct-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:709f100c4b8670f303859fe3e23027d7d988b8c8ce850d17e9f988d0a546a8d1
size 2114801344

3
Llama-Doctor-3.2-3B-Instruct-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:38fd1423e055564e9fa3d37003a62bf9db79acd348a90fa0b051a1f2c9d7cb53
size 2019378880

3
Llama-Doctor-3.2-3B-Instruct-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1bfb10308d5fd33009553e22e5d89b1a80dcdb1c832c94116b8411d8d38d86f5
size 1928201920

3
Llama-Doctor-3.2-3B-Instruct-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7609244efcc81e7565d648d8a6eb501a600f8627fb201c7b414222fa43b5b5cf
size 2417577664

3
Llama-Doctor-3.2-3B-Instruct-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f9d61fa38b5ae6454183b894ef84e338e2029b3a9d63ece52ed77807a288e73
size 2322155200

3
Llama-Doctor-3.2-3B-Instruct-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:834874d866ea28b9fd7a6ce3dd516dd7a205e83d151a28cd2e68c8e104d96710
size 2269513408

3
Llama-Doctor-3.2-3B-Instruct-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:50ba69a4e31379129fb02cbdf2f89832e9e162a5a2ebf07946e8f2a89a16251b
size 2643855040

3
Llama-Doctor-3.2-3B-Instruct-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:003f2efe0dbd411ae782ac4cf2fecbc3d65e136248eadf2f4385949b6ea2b0cf
size 2739277504

3
Llama-Doctor-3.2-3B-Instruct-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ae029f4611ca38a1dc4e86d607c2e893f069d0c92751cf5cc0ad7ca02a562ddd
size 3421900480

3
Llama-Doctor-3.2-3B-Instruct-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:72e1f286bcbcde8a8965bd033288e4ff8aefa65f1c9e6be2b59615f5244887db
size 6433688992

3
Llama-Doctor-3.2-3B-Instruct.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b7836faf619148cdf5389c24952a965a8333475cfe708bd9b8d81fd21067a59e
size 2988390

176
README.md Normal file
View File

@@ -0,0 +1,176 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
datasets:
- avaliev/chat_doctor
tags:
- Llama-3.2
- 3B
- Llama-Doctor
- Instruct
- Llama-Cpp
- meta
- safetensors
license: mit
base_model: prithivMLmods/Llama-Doctor-3.2-3B-Instruct
language:
- en
---
## Llamacpp imatrix Quantizations of Llama-Doctor-3.2-3B-Instruct
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4058">b4058</a> for quantization.
Original model: https://huggingface.co/prithivMLmods/Llama-Doctor-3.2-3B-Instruct
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 July 2024
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Llama-Doctor-3.2-3B-Instruct-f16.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-f16.gguf) | f16 | 6.43GB | false | Full F16 weights. |
| [Llama-Doctor-3.2-3B-Instruct-Q8_0.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q8_0.gguf) | Q8_0 | 3.42GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Llama-Doctor-3.2-3B-Instruct-Q6_K_L.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q6_K_L.gguf) | Q6_K_L | 2.74GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q6_K.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q6_K.gguf) | Q6_K | 2.64GB | false | Very high quality, near perfect, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q5_K_L.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q5_K_L.gguf) | Q5_K_L | 2.42GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q5_K_M.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q5_K_M.gguf) | Q5_K_M | 2.32GB | false | High quality, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q5_K_S.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q5_K_S.gguf) | Q5_K_S | 2.27GB | false | High quality, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_K_L.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_K_L.gguf) | Q4_K_L | 2.11GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_K_M.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_K_M.gguf) | Q4_K_M | 2.02GB | false | Good quality, default size for most use cases, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_K_S.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_K_S.gguf) | Q4_K_S | 1.93GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_0_8_8.gguf) | Q4_0_8_8 | 1.92GB | false | Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). *Don't use on Mac*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_0_4_8.gguf) | Q4_0_4_8 | 1.92GB | false | Optimized for ARM inference. Requires 'i8mm' support (see details below). *Don't use on Mac*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_0_4_4.gguf) | Q4_0_4_4 | 1.92GB | false | Optimized for ARM inference. Should work well on all ARM chips, not for use with GPUs. *Don't use on Mac*. |
| [Llama-Doctor-3.2-3B-Instruct-Q4_0.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q4_0.gguf) | Q4_0 | 1.92GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Llama-Doctor-3.2-3B-Instruct-Q3_K_XL.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q3_K_XL.gguf) | Q3_K_XL | 1.91GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Llama-Doctor-3.2-3B-Instruct-IQ4_XS.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-IQ4_XS.gguf) | IQ4_XS | 1.83GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Llama-Doctor-3.2-3B-Instruct-Q3_K_L.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q3_K_L.gguf) | Q3_K_L | 1.82GB | false | Lower quality but usable, good for low RAM availability. |
| [Llama-Doctor-3.2-3B-Instruct-Q3_K_M.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q3_K_M.gguf) | Q3_K_M | 1.69GB | false | Low quality. |
| [Llama-Doctor-3.2-3B-Instruct-IQ3_M.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-IQ3_M.gguf) | IQ3_M | 1.60GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Llama-Doctor-3.2-3B-Instruct-Q3_K_S.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q3_K_S.gguf) | Q3_K_S | 1.54GB | false | Low quality, not recommended. |
| [Llama-Doctor-3.2-3B-Instruct-IQ3_XS.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-IQ3_XS.gguf) | IQ3_XS | 1.48GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Llama-Doctor-3.2-3B-Instruct-Q2_K_L.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q2_K_L.gguf) | Q2_K_L | 1.46GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Llama-Doctor-3.2-3B-Instruct-Q2_K.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-Q2_K.gguf) | Q2_K | 1.36GB | false | Very low quality but surprisingly usable. |
| [Llama-Doctor-3.2-3B-Instruct-IQ2_M.gguf](https://huggingface.co/bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF/blob/main/Llama-Doctor-3.2-3B-Instruct-IQ2_M.gguf) | IQ2_M | 1.23GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF --include "Llama-Doctor-3.2-3B-Instruct-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Llama-Doctor-3.2-3B-Instruct-GGUF --include "Llama-Doctor-3.2-3B-Instruct-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Llama-Doctor-3.2-3B-Instruct-Q8_0) or download them all in place (./)
</details>
## Q4_0_X_X information
These are *NOT* for Metal (Apple) or GPU (nvidia/AMD/intel) offloading, only ARM chips (and certain AVX2/AVX512 CPUs).
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
If you're using a CPU that supports AVX2 or AVX512 (typically server CPUs and AMD's latest Zen5 CPUs) and are not offloading to a GPU, the Q4_0_8_8 may offer a nice speed as well:
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}