ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-13 11:50:17 +08:00
commit 7eaa8e4d99
32 changed files with 341 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
.gitattributes vendored Normal file
View File

@@ -0,0 +1,64 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
imatrix.dat filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-F16.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
Qwen2.5-Coder-3B-Instruct-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

3
Qwen2.5-Coder-3B-Instruct-F16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b293d7aa8e6396f427a8ff7b8367c1cab3524dc0c92aeb85b2c9f3f7b7ebfeb9
size 6178317280

3
Qwen2.5-Coder-3B-Instruct-IQ1_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4f306a5f55b769bd277edaa3aec205bc32243f2b02fadfd18240728ceced2043
size 850027808

3
Qwen2.5-Coder-3B-Instruct-IQ1_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a219451cfa5f86ed755f00a1b6afc32ce6b97a491ac8549aba40dbc5b8738de
size 791094560

3
Qwen2.5-Coder-3B-Instruct-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7f857de9292150d86865626c69c051b2e4c06fcef981c9cc0641b78dfce2d29c
size 1140516128

3
Qwen2.5-Coder-3B-Instruct-IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2caa14a8067a825669b8e03ca087ba621265f2260cafeadcc37973339130faa7
size 1061938464

3
Qwen2.5-Coder-3B-Instruct-IQ2_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:752c77059e7e637bb1ddf5ce13f67c80837b4aac6214b20863362c3f79438a03
size 1031546144

3
Qwen2.5-Coder-3B-Instruct-IQ2_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d55e5f75a352f1818237ea83dae0f808296d33d1da79be48ad6d2eeb6b2ca536
size 948249888

3
Qwen2.5-Coder-3B-Instruct-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a1b14f8c463fe184180bec62a1eeafa92d07da61887b3254c69970829eca970
size 1488895264

3
Qwen2.5-Coder-3B-Instruct-IQ3_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c45797a02970190f5662638cf5f7cf625166aae42e940c7de3e5cd8e2ab3b68c
size 1456864544

3
Qwen2.5-Coder-3B-Instruct-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:df7fb2cc79da9e5a07c0115eaf35b4f644a63f55776b289eb5cd1eceb5b473b3
size 1391836448

3
Qwen2.5-Coder-3B-Instruct-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:881e7b44c291d432032d453e02fa87d62b3d25952808c0bae0d480721c2b2e55
size 1282827552

3
Qwen2.5-Coder-3B-Instruct-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2eeefc748ccd96d8ae0931c41ff8a8ec52065d0d913d15fc287f4d93a2c2d7fb
size 1825209632

3
Qwen2.5-Coder-3B-Instruct-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e9f89b9ecb366d16f84e802c8055d6a2ce807c52322950b7faf162f894f47d0
size 1739095328

3
Qwen2.5-Coder-3B-Instruct-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cac9204bdf9ddb1d4f096450ef360b8514b216ca01732d8d954afae89353cf4a
size 1274756384

3
Qwen2.5-Coder-3B-Instruct-Q2_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eda340e9999e167483809600b5c18eedebcad556445e3bf469c0d00ae26b0469
size 1198128416

3
Qwen2.5-Coder-3B-Instruct-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8946dc75b62a7bc36a0fa60d51d928b37566bd1e04103f8335034851186da9ea
size 1707392288

3
Qwen2.5-Coder-3B-Instruct-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9efc4a26ca440c4cd0950929962774592e74371f8658e7a6d0a4df393e1b05f3
size 1590476064

3
Qwen2.5-Coder-3B-Instruct-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5c9ff3d31dc73b87456897845de11051a3c0e8702c4961a527108f1eab898420
size 1454357792

3
Qwen2.5-Coder-3B-Instruct-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f28ede881bf33af1a16accf6793fc6d281834acabf1369a37e02ce56485fb30f
size 1828486432

3
Qwen2.5-Coder-3B-Instruct-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d0b0bcfdd493ac024b158abdc7db99a56a3ce96d4f50056d55cb1bfe848c9425
size 1996258592

3
Qwen2.5-Coder-3B-Instruct-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d999adc0c4fea41083a485b139aa7cba72f9d7d9090bc335ca9e575cc1e47868
size 1929903392

3
Qwen2.5-Coder-3B-Instruct-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d41c420f26361920e1ec2f1c0c7062cf17cafe14c50471ee4bee3a2c6791b1fd
size 1834384672

3
Qwen2.5-Coder-3B-Instruct-Q5_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:99c1dd5c8310c634912ecb2122911066f90b9dfc0a55629407a393ad3ed45f3a
size 2175302944

3
Qwen2.5-Coder-3B-Instruct-Q5_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:17dcaf7aa412a2576e95d4a15b84edc23b2e9b5442935553d970cd72069bdc3d
size 2343075104

3
Qwen2.5-Coder-3B-Instruct-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c950908419914d8fdfdec563c6e96d7af82aa2ff8378853683f6f448be84c374
size 2224815392

3
Qwen2.5-Coder-3B-Instruct-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a51bc08c741dae197df10b8f7061364b4c7c8ec1e8247e2cc48dfce9ee7ba5fe
size 2169666848

3
Qwen2.5-Coder-3B-Instruct-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ab3477cb3cd90a9caf2aaad707aff8b0c34658fa34a4a526b83428ffa7e4104b
size 2538159392

3
Qwen2.5-Coder-3B-Instruct-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:db2c0ef69c934c71705a8e955b7dc02df517a1b80723a056ab4e98561698452a
size 3285476640

160
README.md Normal file
View File

@@ -0,0 +1,160 @@
---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE
language:
- en
base_model:
- Qwen/Qwen2.5-Coder-3B
pipeline_tag: text-generation
library_name: transformers
tags:
- code
- codeqwen
- chat
- qwen
- qwen-coder
---
<br><img src="https://cdn-uploads.huggingface.co/production/uploads/646410e04bf9122922289dc7/wHRc2ZKSErOs92ZcdKQrY.webp" width="720"><br>
# Llama.cpp imatrix quantizations of [Qwen/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct)
Using llama.cpp commit [3ad5451](https://github.com/ggerganov/llama.cpp/commit/3ad5451) for quantization.
All quants were made using the imatrix option and Bartowski's [calibration file](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8).
<hr>
# Perplexity table (the lower the better)
| Quant | Size (MB) | PPL | Size (%) | Accuracy (%) | PPL error rate |
| -------------------------------------------------------------------------------------------------------------------------------- | --------- | ------- | -------- | ------------ | -------------- |
| [IQ1_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ1_S.gguf) | 754 | 38.2733 | 12.80 | 25.21 | 0.68 |
| [IQ1_M](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ1_M.gguf) | 810 | 23.9542 | 13.75 | 40.28 | 0.41 |
| [IQ2_XXS](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ2_XXS.gguf) | 904 | 22.7872 | 15.34 | 42.34 | 0.41 |
| [IQ2_XS](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ2_XS.gguf) | 983 | 14.4025 | 16.68 | 66.99 | 0.24 |
| [IQ2_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ2_S.gguf) | 1012 | 13.6118 | 17.17 | 70.88 | 0.22 |
| [IQ2_M](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ2_M.gguf) | 1087 | 12.6560 | 18.45 | 76.24 | 0.20 |
| [Q2_K_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q2_K_S.gguf) | 1142 | 12.4745 | 19.38 | 77.35 | 0.20 |
| [Q2_K](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q2_K.gguf) | 1215 | 11.5168 | 20.62 | 83.78 | 0.18 |
| [IQ3_XXS](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ3_XXS.gguf) | 1223 | 10.8327 | 20.76 | 89.07 | 0.17 |
| [IQ3_XS](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ3_XS.gguf) | 1327 | 10.2693 | 22.52 | 93.96 | 0.16 |
| [Q3_K_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q3_K_S.gguf) | 1386 | 32.8121 | 23.52 | 29.41 | 0.62 |
| [IQ3_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ3_S.gguf) | 1389 | 10.2409 | 23.57 | 94.22 | 0.16 |
| [IQ3_M](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ3_M.gguf) | 1419 | 10.1950 | 24.08 | 94.64 | 0.16 |
| [Q3_K_M](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q3_K_M.gguf) | 1516 | 20.7076 | 25.73 | 46.59 | 0.36 |
| [Q3_K_L](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q3_K_L.gguf) | 1628 | 20.6384 | 27.63 | 46.75 | 0.36 |
| [IQ4_XS](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ4_XS.gguf) | 1658 | 9.9013 | 28.14 | 97.45 | 0.15 |
| [IQ4_NL](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-IQ4_NL.gguf) | 1740 | 9.8926 | 29.53 | 97.53 | 0.15 |
| [Q4_0](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q4_0.gguf) | 1743 | 10.0757 | 29.58 | 95.76 | 0.16 |
| [Q4_K_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q4_K_S.gguf) | 1749 | 9.8497 | 29.68 | 97.96 | 0.15 |
| [Q4_K_M](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q4_K_M.gguf) | 1840 | 9.8160 | 31.23 | 98.29 | 0.15 |
| [Q4_1](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q4_1.gguf) | 1903 | 9.8176 | 32.30 | 98.28 | 0.15 |
| [Q5_K_S](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q5_K_S.gguf) | 2069 | 9.7456 | 35.12 | 99.00 | 0.15 |
| [Q5_0](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q5_0.gguf) | 2074 | 9.7237 | 35.20 | 99.23 | 0.15 |
| [Q5_K_M](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q5_K_M.gguf) | 2121 | 9.7331 | 36.00 | 99.13 | 0.15 |
| [Q5_1](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q5_1.gguf) | 2234 | 9.7260 | 37.92 | 99.20 | 0.15 |
| [Q6_K](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q6_K.gguf) | 2420 | 9.6939 | 41.07 | 99.53 | 0.15 |
| [Q8_0](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-Q8_0.gguf) | 3133 | 9.6710 | 53.17 | 99.77 | 0.15 |
| [F16](https://huggingface.co/ThomasBaruzier/Qwen2.5-Coder-3B-Instruct-GGUF/blob/main/Qwen2.5-Coder-3B-Instruct-F16.gguf) | 5892 | 9.6486 | 100 | 100 | 0.15 |
<hr>
# Qwen2.5-Coder-3B-Instruct
<a href="https://chat.qwenlm.ai/" target="_blank" style="margin: 2px;">
<img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
</a>
## Introduction
Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). As of now, Qwen2.5-Coder has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters, to meet the needs of different developers. Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:
- Significantly improvements in **code generation**, **code reasoning** and **code fixing**. Base on the strong Qwen2.5, we scale up the training tokens into 5.5 trillion including source code, text-code grounding, Synthetic data, etc. Qwen2.5-Coder-32B has become the current state-of-the-art open-source codeLLM, with its coding abilities matching those of GPT-4o.
- A more comprehensive foundation for real-world applications such as **Code Agents**. Not only enhancing coding capabilities but also maintaining its strengths in mathematics and general competencies.
**This repo contains the instruction-tuned 3B Qwen2.5-Coder model**, which has the following features:
- Type: Causal Language Models
- Training Stage: Pretraining & Post-training
- Architecture: transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
- Number of Parameters: 3.09B
- Number of Paramaters (Non-Embedding): 2.77B
- Number of Layers: 36
- Number of Attention Heads (GQA): 16 for Q and 2 for KV
- Context Length: Full 32,768 tokens
For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5-coder-family/), [GitHub](https://github.com/QwenLM/Qwen2.5-Coder), [Documentation](https://qwen.readthedocs.io/en/latest/), [Arxiv](https://arxiv.org/abs/2409.12186).
## Requirements
The code of Qwen2.5-Coder has been in the latest Hugging face `transformers` and we advise you to use the latest version of `transformers`.
With `transformers<4.37.0`, you will encounter the following error:
```
KeyError: 'qwen2'
```
## Quickstart
Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Qwen/Qwen2.5-Coder-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "write a quick sort algorithm."
messages = [
{"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=512
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```
## Evaluation & Performance
Detailed evaluation results are reported in this [📑 blog](https://qwenlm.github.io/blog/qwen2.5-coder-family/).
For requirements on GPU memory and the respective throughput, see results [here](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html).
## Citation
If you find our work helpful, feel free to give us a cite.
```
@article{hui2024qwen2,
title={Qwen2. 5-Coder Technical Report},
author={Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and others},
journal={arXiv preprint arXiv:2409.12186},
year={2024}
}
@article{qwen2,
title={Qwen2 Technical Report},
author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
journal={arXiv preprint arXiv:2407.10671},
year={2024}
}
```

3
imatrix.dat Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ccca27be8de5ef40cf627cafb9ac55af300f4e02e763f8b0eb18e3ceafd1d707
size 3363008

30
perplexity.md Normal file
View File

@@ -0,0 +1,30 @@
| Quant | Size (MB) | PPL | Size (%) | Accuracy (%) | PPL error rate |
| ------ | --------- | ------- | -------- | ------------ | -------------- |
| IQ1_S | 754 | 38.2733 | 12.80 | 25.21 | 0.68 |
| IQ1_M | 810 | 23.9542 | 13.75 | 40.28 | 0.41 |
| IQ2_XXS | 904 | 22.7872 | 15.34 | 42.34 | 0.41 |
| IQ2_XS | 983 | 14.4025 | 16.68 | 66.99 | 0.24 |
| IQ2_S | 1012 | 13.6118 | 17.17 | 70.88 | 0.22 |
| IQ2_M | 1087 | 12.6560 | 18.45 | 76.24 | 0.20 |
| Q2_K_S | 1142 | 12.4745 | 19.38 | 77.35 | 0.20 |
| Q2_K | 1215 | 11.5168 | 20.62 | 83.78 | 0.18 |
| IQ3_XXS | 1223 | 10.8327 | 20.76 | 89.07 | 0.17 |
| IQ3_XS | 1327 | 10.2693 | 22.52 | 93.96 | 0.16 |
| Q3_K_S | 1386 | 32.8121 | 23.52 | 29.41 | 0.62 |
| IQ3_S | 1389 | 10.2409 | 23.57 | 94.22 | 0.16 |
| IQ3_M | 1419 | 10.1950 | 24.08 | 94.64 | 0.16 |
| Q3_K_M | 1516 | 20.7076 | 25.73 | 46.59 | 0.36 |
| Q3_K_L | 1628 | 20.6384 | 27.63 | 46.75 | 0.36 |
| IQ4_XS | 1658 | 9.9013 | 28.14 | 97.45 | 0.15 |
| IQ4_NL | 1740 | 9.8926 | 29.53 | 97.53 | 0.15 |
| Q4_0 | 1743 | 10.0757 | 29.58 | 95.76 | 0.16 |
| Q4_K_S | 1749 | 9.8497 | 29.68 | 97.96 | 0.15 |
| Q4_K_M | 1840 | 9.8160 | 31.23 | 98.29 | 0.15 |
| Q4_1 | 1903 | 9.8176 | 32.30 | 98.28 | 0.15 |
| Q5_K_S | 2069 | 9.7456 | 35.12 | 99.00 | 0.15 |
| Q5_0 | 2074 | 9.7237 | 35.20 | 99.23 | 0.15 |
| Q5_K_M | 2121 | 9.7331 | 36.00 | 99.13 | 0.15 |
| Q5_1 | 2234 | 9.7260 | 37.92 | 99.20 | 0.15 |
| Q6_K | 2420 | 9.6939 | 41.07 | 99.53 | 0.15 |
| Q8_0 | 3133 | 9.6710 | 53.17 | 99.77 | 0.15 |
| F16 | 5892 | 9.6486 | 100 | 100 | 0.15 |