bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-19 01:07:13 +08:00
commit 97d8c0c92a
24 changed files with 233 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
.gitattributes vendored Normal file
View File

@@ -0,0 +1,56 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview-bf16.gguf filter=lfs diff=lfs merge=lfs -text
Replete-LLM-Qwen2-7b_Beta-Preview.imatrix filter=lfs diff=lfs merge=lfs -text

113
README.md Normal file
View File

@@ -0,0 +1,113 @@
---
base_model: Replete-AI/Replete-LLM-Qwen2-7b_Beta-Preview
datasets:
- Replete-AI/Everything_Instruct_8k_context_filtered
language:
- en
license: apache-2.0
pipeline_tag: text-generation
tags:
- unsloth
quantized_by: bartowski
---
## Llamacpp imatrix Quantizations of Replete-LLM-Qwen2-7b_Beta-Preview
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3460">b3460</a> for quantization.
Original model: https://huggingface.co/Replete-AI/Replete-LLM-Qwen2-7b_Beta-Preview
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<|endoftext|>
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Replete-LLM-Qwen2-7b_Beta-Preview-bf16.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-bf16.gguf) | bf16 | 15.24GB | false | Full BF16 weights. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0.gguf) | Q8_0 | 8.10GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K_L.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K_L.gguf) | Q6_K_L | 6.52GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K.gguf) | Q6_K | 6.25GB | false | Very high quality, near perfect, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_L.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_L.gguf) | Q5_K_L | 5.78GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_M.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_M.gguf) | Q5_K_M | 5.44GB | false | High quality, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_S.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_S.gguf) | Q5_K_S | 5.32GB | false | High quality, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_L.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_L.gguf) | Q4_K_L | 5.09GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_M.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_M.gguf) | Q4_K_M | 4.68GB | false | Good quality, default size for must use cases, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_XL.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_XL.gguf) | Q3_K_XL | 4.57GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_S.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_S.gguf) | Q4_K_S | 4.46GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-IQ4_XS.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-IQ4_XS.gguf) | IQ4_XS | 4.22GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_L.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_L.gguf) | Q3_K_L | 4.09GB | false | Lower quality but usable, good for low RAM availability. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_M.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_M.gguf) | Q3_K_M | 3.81GB | false | Low quality. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_M.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_M.gguf) | IQ3_M | 3.57GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K_L.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K_L.gguf) | Q2_K_L | 3.55GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_S.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_S.gguf) | Q3_K_S | 3.49GB | false | Low quality, not recommended. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_XS.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_XS.gguf) | IQ3_XS | 3.35GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K.gguf) | Q2_K | 3.02GB | false | Very low quality but surprisingly usable. |
| [Replete-LLM-Qwen2-7b_Beta-Preview-IQ2_M.gguf](https://huggingface.co/bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF/blob/main/Replete-LLM-Qwen2-7b_Beta-Preview-IQ2_M.gguf) | IQ2_M | 2.78GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF --include "Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Replete-LLM-Qwen2-7b_Beta-Preview-GGUF --include "Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0.gguf/*" --local-dir Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0
```
You can either specify a new local-dir (Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0) or download them all in place (./)
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

3
Replete-LLM-Qwen2-7b_Beta-Preview-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2a3195b4f19c631768eff0a6593791862518cb3dcd92296968caad65d07fe2f0
size 2780340480

3
Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c3c48ab7ffa6bff4d2011eae84a8e25d3d17468df789d1cae67e4c2c9307a4e1
size 3574010112

3
Replete-LLM-Qwen2-7b_Beta-Preview-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:20d2cbea450fcd0f4c95c409432ea869fa0477a17d3ac0cf013ad84cb02bbae3
size 3346254080

3
Replete-LLM-Qwen2-7b_Beta-Preview-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9171365f1e2eb17ea0b776edc89ae1366b09616bfd9abc3e5e39dc2d5b3287d
size 4218470656

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:09aecb104a4d506527a1bef4d1c8073b353deb8321f3849b965363c5fa34d7be
size 3015938304

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e8812528427e7e56288a10b63078fadf5ad084edf51324b7ece00bde2fcc9ff1
size 3548162304

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:71b68cd654afba3c71599f3c3844859d8428134e4f698c372d6e96d8ca7dc98e
size 4088457472

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f82cc11c2c340b01e390e6bd2eb310c7398cf9e559e4d44a8a199076689fc0f9
size 3808389376

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1815cdc8fca1b5f6a1cfa718e57de002c90a09164f8b46a2f080e38d6d4e0449
size 3492366592

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2e03a832ca48edf986076a7f432ae3bbfd8aa1eb56e9f44318860613dbb96523
size 4565330176

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d90f7f02ed218d4f8918f5a110aa5b58193fe2f09359d6ba30bcd2466793ee37
size 5087561984

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6f86511947c45c46a96ea0784908083b5d66b7361fa6a88cc8f1e722f65c60ef
size 4683071744

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ce3afb7d63b4a54e4fe202562027f587def0662d0d5092727e953f421d94cea2
size 4457767168

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e665c3bbe2626be7adc88aad8b7553528c9c649c6e30f63054b7c27ca027d82
size 5781195008

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:efa4ffc9f5c024092f1cb2aa644a98f440bdfed14532dbe0dc282a672b364ae6
size 5444829440

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:431e0111bca69a25299e0ebd89d0139b42487da478aa664420cf9081b5c73713
size 5315174656

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c2f419d34d5fdd631acf2600411aa7e9092aaf30daba5b49de780621fb8f9fc
size 6254196992

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:45c29b94e3bc832f573ceeb72415daae23e516346528819a73225cc9ffb3b0de
size 6518180096

3
Replete-LLM-Qwen2-7b_Beta-Preview-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:449dcf0857845c2dac940f2327f0fed419b23ab8383db841b35d4bf9f4cc3b78
size 8098523392

3
Replete-LLM-Qwen2-7b_Beta-Preview-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4981bb1efaf6e77c074f5681a72565d3ae548e23169152625fba52c6383c755d
size 15237851104

3
Replete-LLM-Qwen2-7b_Beta-Preview.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c85afc5a206c86238220c32ae02e4f7976794cb037232c4be55bed33f524d95
size 4536678

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}