bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-27 03:23:13 +08:00
commit f6fb714724
28 changed files with 287 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
.gitattributes vendored Normal file
View File

@@ -0,0 +1,47 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

3
MetaStoneTec_MetaStone-L1-7B-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c188d9950d043f4c10de1d5a6ec388813271ae99d8f4f7fd5addafb62020bcd
size 2780342208

3
MetaStoneTec_MetaStone-L1-7B-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:462220f7c1bed83796949e7da77a784cfc3f69083cd23051c5d1a58417cce660
size 3574011840

3
MetaStoneTec_MetaStone-L1-7B-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c124060dd217b75bb407a3456634b4855e810fc89328ac1c59a8964b11c9ddd0
size 3346255808

3
MetaStoneTec_MetaStone-L1-7B-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc89590d79ad938b5f1aa111c3b493b4cb8269fb30f494e0d9d10695b7154c5c
size 3114514368

3
MetaStoneTec_MetaStone-L1-7B-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:075f8a1a79d2c47478c10356e9aca0d9bdc70e364bc2422aecdac10ce989be6b
size 4437813184

3
MetaStoneTec_MetaStone-L1-7B-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:220168383e9bc62b3d0f3449c52cd358489683fb43e2bd60fc6157fb76f1f47b
size 4218472384

3
MetaStoneTec_MetaStone-L1-7B-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2297ab91f90530faeed1aacd5cbc598155c2f67d0c64bc8d03e1c0b1a198ec3a
size 3015940032

3
MetaStoneTec_MetaStone-L1-7B-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:beccbc1354b21cca1ecfdd3ae2c7d208934908695db172e3e8f83bcc768deb7f
size 3548164032

3
MetaStoneTec_MetaStone-L1-7B-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:840373eef59b8ef92d1c51334f5ee0d4092e2a19c9154e36395578964c231418
size 4088459200

3
MetaStoneTec_MetaStone-L1-7B-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:993c266d1feee2946159ed198cd8602f04aa2e9b07894f19154c04ff381b5054
size 3808391104

3
MetaStoneTec_MetaStone-L1-7B-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b0808fc3a0ca9848e9d1df457073333635a015b29462d7cfb1ec24ee16bcd52b
size 3492368320

3
MetaStoneTec_MetaStone-L1-7B-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3368965592cd722fa4b2a80666fdf5b1e4d94f5be6c23fa01728ac4e912e248a
size 4565331904

3
MetaStoneTec_MetaStone-L1-7B-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7cf7b9ed678d8252bbe1001579255904a15fbbf52e708f9a78f7f41d05e7f2fa
size 4444121024

3
MetaStoneTec_MetaStone-L1-7B-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2e0f751c6497b91113b8a415449e682e5c91f8529909a30f20429a4498235361
size 4873283520

3
MetaStoneTec_MetaStone-L1-7B-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2862f42eea46873396ed5c01d82a4578d4785bbaf2e44c61c98c020fbcdb5c3d
size 5087563712

3
MetaStoneTec_MetaStone-L1-7B-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b5315f053ad3ebdfb389d7488cbbfab9d1d5e8dd625e83ad5aa78718a55045a5
size 4683073472

3
MetaStoneTec_MetaStone-L1-7B-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bf5723e3de25767fef85de7cff9eb9365720e94672e5ffaacd7e072bdcdf7ca9
size 4457768896

3
MetaStoneTec_MetaStone-L1-7B-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:498a87286a24db126cb91e4a676b14001f4ba0c7cf7772baad71816b28692169
size 5781196736

3
MetaStoneTec_MetaStone-L1-7B-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4a516777091b7f86e295ea357d7a4128eae4ddbc72e1b43374e7bb45fc9cf915
size 5444831168

3
MetaStoneTec_MetaStone-L1-7B-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c67e91b7975189d1002e40a8aa33dd7e0459f1d034e42ceb68be3eaeab6c6d84
size 5315176384

3
MetaStoneTec_MetaStone-L1-7B-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:66a27902e2e01ca2cb2178825df02d507ec632f84b8f84a3a083ca4eb2fefefe
size 6254198720

3
MetaStoneTec_MetaStone-L1-7B-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:02b6ae1473c7588a84d6417ce4afa206199923e630a5d01097206aa491ba2184
size 6518181824

3
MetaStoneTec_MetaStone-L1-7B-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54eb0f2cd0d0fff3bcf552164c2d363050251ac5e333f7fb448d1858f5f5f5ca
size 8098525120

3
MetaStoneTec_MetaStone-L1-7B-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:72911cc93ee56823d8ef34cc176c38dc802a5bccc4536f29afb4d00ddbd0dfd0
size 15237852832

BIN
MetaStoneTec_MetaStone-L1-7B.imatrix Normal file
View File

Binary file not shown.

167
README.md Normal file
View File

@@ -0,0 +1,167 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model: MetaStoneTec/MetaStone-L1-7B
---
## Llamacpp imatrix Quantizations of MetaStone-L1-7B by MetaStoneTec
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4882">b4882</a> for quantization.
Original model: https://huggingface.co/MetaStoneTec/MetaStone-L1-7B
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
Run them directly with [llama.cpp](https://github.com/ggerganov/llama.cpp), or any other llama.cpp based project
## Prompt format
```
<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜><｜end▁of▁sentence｜><｜Assistant｜><think>
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [MetaStone-L1-7B-bf16.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-bf16.gguf) | bf16 | 15.24GB | false | Full BF16 weights. |
| [MetaStone-L1-7B-Q8_0.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q8_0.gguf) | Q8_0 | 8.10GB | false | Extremely high quality, generally unneeded but max available quant. |
| [MetaStone-L1-7B-Q6_K_L.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q6_K_L.gguf) | Q6_K_L | 6.52GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [MetaStone-L1-7B-Q6_K.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q6_K.gguf) | Q6_K | 6.25GB | false | Very high quality, near perfect, *recommended*. |
| [MetaStone-L1-7B-Q5_K_L.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q5_K_L.gguf) | Q5_K_L | 5.78GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [MetaStone-L1-7B-Q5_K_M.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q5_K_M.gguf) | Q5_K_M | 5.44GB | false | High quality, *recommended*. |
| [MetaStone-L1-7B-Q5_K_S.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q5_K_S.gguf) | Q5_K_S | 5.32GB | false | High quality, *recommended*. |
| [MetaStone-L1-7B-Q4_K_L.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q4_K_L.gguf) | Q4_K_L | 5.09GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [MetaStone-L1-7B-Q4_1.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q4_1.gguf) | Q4_1 | 4.87GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [MetaStone-L1-7B-Q4_K_M.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q4_K_M.gguf) | Q4_K_M | 4.68GB | false | Good quality, default size for most use cases, *recommended*. |
| [MetaStone-L1-7B-Q3_K_XL.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q3_K_XL.gguf) | Q3_K_XL | 4.57GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [MetaStone-L1-7B-Q4_K_S.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q4_K_S.gguf) | Q4_K_S | 4.46GB | false | Slightly lower quality with more space savings, *recommended*. |
| [MetaStone-L1-7B-Q4_0.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q4_0.gguf) | Q4_0 | 4.44GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [MetaStone-L1-7B-IQ4_NL.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-IQ4_NL.gguf) | IQ4_NL | 4.44GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [MetaStone-L1-7B-IQ4_XS.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-IQ4_XS.gguf) | IQ4_XS | 4.22GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [MetaStone-L1-7B-Q3_K_L.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q3_K_L.gguf) | Q3_K_L | 4.09GB | false | Lower quality but usable, good for low RAM availability. |
| [MetaStone-L1-7B-Q3_K_M.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q3_K_M.gguf) | Q3_K_M | 3.81GB | false | Low quality. |
| [MetaStone-L1-7B-IQ3_M.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-IQ3_M.gguf) | IQ3_M | 3.57GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [MetaStone-L1-7B-Q2_K_L.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q2_K_L.gguf) | Q2_K_L | 3.55GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [MetaStone-L1-7B-Q3_K_S.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q3_K_S.gguf) | Q3_K_S | 3.49GB | false | Low quality, not recommended. |
| [MetaStone-L1-7B-IQ3_XS.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-IQ3_XS.gguf) | IQ3_XS | 3.35GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [MetaStone-L1-7B-IQ3_XXS.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-IQ3_XXS.gguf) | IQ3_XXS | 3.11GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [MetaStone-L1-7B-Q2_K.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-Q2_K.gguf) | Q2_K | 3.02GB | false | Very low quality but surprisingly usable. |
| [MetaStone-L1-7B-IQ2_M.gguf](https://huggingface.co/bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF/blob/main/MetaStoneTec_MetaStone-L1-7B-IQ2_M.gguf) | IQ2_M | 2.78GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF --include "MetaStoneTec_MetaStone-L1-7B-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/MetaStoneTec_MetaStone-L1-7B-GGUF --include "MetaStoneTec_MetaStone-L1-7B-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (MetaStoneTec_MetaStone-L1-7B-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}