bartowski/Menlo_Lucy-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Menlo_Lucy-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-30 18:03:14 +08:00
commit 4e3cd5201b
27 changed files with 293 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
.gitattributes vendored Normal file
View File

@@ -0,0 +1,49 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Menlo_Lucy.imatrix filter=lfs diff=lfs merge=lfs -text

3
Menlo_Lucy-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:52aa49ff8a7b9dbebb04cc527b0d9faf12a642f2673e21eea7179d485c95605d
size 895662336

3
Menlo_Lucy-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6fcd320e8ff8ee0d6df69f887954ee5c4193efe10acf3f14fa9c0ce776986b7f
size 834222336

3
Menlo_Lucy-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8d258325d003a6ddf6e7d4fa88d771ca2da0a1140505a13109966301f39c5550
size 754360576

3
Menlo_Lucy-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:58b8f21bbf77ba033cbdb5ad6913cfc6f4b07587c028bcfc61a6a56c13788fad
size 1054423296

3
Menlo_Lucy-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:084c2eb1f1a57db53deed2966903088b24f3abd43cf752eb808d106e13067a95
size 1010383104

3
Menlo_Lucy-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:828e60220d03dfafd0cc535ea5da169b48d08c678f45f21cd7694488d51929cb
size 777795840

3
Menlo_Lucy-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c7c5b5dbb913c526048f2626a65987f2d5002eef6eb895abc513d58e57e87c86
size 853156096

3
Menlo_Lucy-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea25e375b9fc059c5862e33ea643e6436f78811f3c78d3f669a74077c10c84b3
size 1003501824

3
Menlo_Lucy-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ccbe16c64e5ea2be006b2cb392a187f574f67f6d30eaf490ea1d38cd7c320d40
size 939538688

3
Menlo_Lucy-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:713127d7fcce7c2ec100f7075947dc3aa7934b4f3f68b1e65e2f5e5b49e78e2e
size 867252480

3
Menlo_Lucy-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a451a71c70c735fe92298942188655ea0df09570bff0b17a6b6b17f079dabfe4
size 1078862080

3
Menlo_Lucy-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9cca90a3e4e558f1b9f31a4eff9b0bbe4e3b5edd3e8c3bb2e94e674b33b82bbd
size 1056782592

3
Menlo_Lucy-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:62f1f9e6de97668bf84cb03efdeb6688deda35e32c7f87c2e2886ea616eba1c4
size 1142503680

3
Menlo_Lucy-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d1e36862a0b3f8103b63982b9c85ab06c135334632d6c07c482267b41fed1871
size 1182769408

3
Menlo_Lucy-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1cb1682a9dbea9a1c8406721695f3faf6a212554d283585f2ec4608921f7c8b7
size 1107409152

3
Menlo_Lucy-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5bfd50775505bf4973bb71d2ca5cb2055d5526d89957bc4271e4c978a7d3081d
size 1060190464

3
Menlo_Lucy-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b5dbb7d460fabade0749e1e117e141ea45be62b10fefa6e3c98f2585088333ec
size 1333240064

3
Menlo_Lucy-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a3b4ab6d49b3cfaba6c87b91baf38463cc6ea275859c09add1739f00182853ac
size 1257879808

3
Menlo_Lucy-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ce413aff07e9dbc522028b70a1c8641d4d400b7adc8b360e3c87e997323d7be5
size 1230584064

3
Menlo_Lucy-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:10b44e2118c4411cd84dc6c6b0c737a53bc22e3fae1d006aab10765340a36744
size 1417754880

3
Menlo_Lucy-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d63ab222dccbd0ebebd8a071b255ce6a64f6964dfe4f425532877dfe86f7e152
size 1493115136

3
Menlo_Lucy-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0e888cd21367cfd7700e020faa85a6d546b0ca9359933b1e53dcaf92e79f14a1
size 1834426624

3
Menlo_Lucy-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0d733d788d845589a29689e8e344e667fafb08ca12897598a916295066c54d04
size 3447349248

3
Menlo_Lucy.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:26a01efa4201b2055b61daeb5fe871bfb294270a5af87f661e206f3e69cdd371
size 2070886

171
README.md Normal file
View File

@@ -0,0 +1,171 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model: Menlo/Lucy
base_model_relation: quantized
---
## Llamacpp imatrix Quantizations of Lucy by Menlo
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b5924">b5924</a> for quantization.
Original model: https://huggingface.co/Menlo/Lucy
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
Run them directly with [llama.cpp](https://github.com/ggerganov/llama.cpp), or any other llama.cpp based project
## Prompt format
No chat template specified so default is used. This may be incorrect, check original model card for details.
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Lucy-bf16.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-bf16.gguf) | bf16 | 3.45GB | false | Full BF16 weights. |
| [Lucy-Q8_0.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q8_0.gguf) | Q8_0 | 1.83GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Lucy-Q6_K_L.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q6_K_L.gguf) | Q6_K_L | 1.49GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Lucy-Q6_K.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q6_K.gguf) | Q6_K | 1.42GB | false | Very high quality, near perfect, *recommended*. |
| [Lucy-Q5_K_L.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q5_K_L.gguf) | Q5_K_L | 1.33GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Lucy-Q5_K_M.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q5_K_M.gguf) | Q5_K_M | 1.26GB | false | High quality, *recommended*. |
| [Lucy-Q5_K_S.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q5_K_S.gguf) | Q5_K_S | 1.23GB | false | High quality, *recommended*. |
| [Lucy-Q4_K_L.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q4_K_L.gguf) | Q4_K_L | 1.18GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Lucy-Q4_1.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q4_1.gguf) | Q4_1 | 1.14GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [Lucy-Q4_K_M.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q4_K_M.gguf) | Q4_K_M | 1.11GB | false | Good quality, default size for most use cases, *recommended*. |
| [Lucy-Q3_K_XL.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q3_K_XL.gguf) | Q3_K_XL | 1.08GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Lucy-Q4_K_S.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q4_K_S.gguf) | Q4_K_S | 1.06GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Lucy-Q4_0.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q4_0.gguf) | Q4_0 | 1.06GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [Lucy-IQ4_NL.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-IQ4_NL.gguf) | IQ4_NL | 1.05GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [Lucy-IQ4_XS.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-IQ4_XS.gguf) | IQ4_XS | 1.01GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Lucy-Q3_K_L.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q3_K_L.gguf) | Q3_K_L | 1.00GB | false | Lower quality but usable, good for low RAM availability. |
| [Lucy-Q3_K_M.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q3_K_M.gguf) | Q3_K_M | 0.94GB | false | Low quality. |
| [Lucy-IQ3_M.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-IQ3_M.gguf) | IQ3_M | 0.90GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Lucy-Q3_K_S.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q3_K_S.gguf) | Q3_K_S | 0.87GB | false | Low quality, not recommended. |
| [Lucy-Q2_K_L.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q2_K_L.gguf) | Q2_K_L | 0.85GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Lucy-IQ3_XS.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-IQ3_XS.gguf) | IQ3_XS | 0.83GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Lucy-Q2_K.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-Q2_K.gguf) | Q2_K | 0.78GB | false | Very low quality but surprisingly usable. |
| [Lucy-IQ3_XXS.gguf](https://huggingface.co/bartowski/Menlo_Lucy-GGUF/blob/main/Menlo_Lucy-IQ3_XXS.gguf) | IQ3_XXS | 0.75GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Menlo_Lucy-GGUF --include "Menlo_Lucy-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Menlo_Lucy-GGUF --include "Menlo_Lucy-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Menlo_Lucy-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}