初始化项目，由ModelHub XC社区提供模型

Model: legraphista/RoLlama2-7b-Instruct-IMat-GGUF Source: Original Platform
2026-07-25 12:14:10 +08:00
commit 7a1d1dd298
30 changed files with 3539 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,61 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.FP16.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.BF16.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+imatrix.dat filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q3_K.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q4_K.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+RoLlama2-7b-Instruct.IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,97 @@
+---
+base_model: OpenLLM-Ro/RoLlama2-7b-Instruct
+inference: false
+language:
+- ro
+library_name: gguf
+license: cc-by-nc-4.0
+pipeline_tag: text-generation
+quantized_by: legraphista
+tags:
+- quantized
+- GGUF
+- imatrix
+- quantization
+---
+
+# RoLlama2-7b-Instruct-IMat-GGUF
+_Llama.cpp imatrix quantization of RoLlama2-7b-Instruct-IMat-GGUF_
+
+Original Model: [OpenLLM-Ro/RoLlama2-7b-Instruct](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Instruct)  
+Original dtype: `FP32` (`float32`)  
+Quantized by: llama.cpp [b2998](https://github.com/ggerganov/llama.cpp/releases/tag/b2998)  
+IMatrix dataset: [here](https://gist.githubusercontent.com/legraphista/d6d93f1a254bcfc58e0af3777eaec41e/raw/d380e7002cea4a51c33fffd47db851942754e7cc/imatrix.calibration.medium.raw)  
+
+## Files
+
+### IMatrix
+Status: ✅ Available  
+Link: [here](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/imatrix.dat) 
+
+### Common Quants
+| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
+| -------- | ---------- | --------- | ------ | ------------ | -------- |
+| [RoLlama2-7b-Instruct.Q8_0.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q8_0.gguf) | Q8_0 | 7.16GB | ✅ Available | ⚪ No | 📦 No
+| [RoLlama2-7b-Instruct.Q6_K.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q6_K.gguf) | Q6_K | 5.53GB | ✅ Available | ⚪ No | 📦 No
+| [RoLlama2-7b-Instruct.Q4_K.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q4_K.gguf) | Q4_K | 4.08GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.Q3_K.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q3_K.gguf) | Q3_K | 3.30GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.Q2_K.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q2_K.gguf) | Q2_K | 2.53GB | ✅ Available | 🟢 Yes | 📦 No
+
+
+### All Quants
+| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
+| -------- | ---------- | --------- | ------ | ------------ | -------- |
+| [RoLlama2-7b-Instruct.FP16.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.FP16.gguf) | F16 | 13.48GB | ✅ Available | ⚪ No | 📦 No
+| [RoLlama2-7b-Instruct.BF16.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.BF16.gguf) | BF16 | 13.48GB | ✅ Available | ⚪ No | 📦 No
+| [RoLlama2-7b-Instruct.Q5_K.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q5_K.gguf) | Q5_K | 4.78GB | ✅ Available | ⚪ No | 📦 No
+| [RoLlama2-7b-Instruct.Q5_K_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q5_K_S.gguf) | Q5_K_S | 4.65GB | ✅ Available | ⚪ No | 📦 No
+| [RoLlama2-7b-Instruct.Q4_K_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q4_K_S.gguf) | Q4_K_S | 3.86GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.Q3_K_L.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q3_K_L.gguf) | Q3_K_L | 3.60GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.Q3_K_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q3_K_S.gguf) | Q3_K_S | 2.95GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.Q2_K_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.Q2_K_S.gguf) | Q2_K_S | 2.32GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ4_NL.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ4_NL.gguf) | IQ4_NL | 3.83GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ4_XS.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ4_XS.gguf) | IQ4_XS | 3.62GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ3_M.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ3_M.gguf) | IQ3_M | 3.11GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ3_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ3_S.gguf) | IQ3_S | 2.95GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ3_XS.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ3_XS.gguf) | IQ3_XS | 2.80GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ3_XXS.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ3_XXS.gguf) | IQ3_XXS | 2.59GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ2_M.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ2_M.gguf) | IQ2_M | 2.36GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ2_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ2_S.gguf) | IQ2_S | 2.20GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ2_XS.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ2_XS.gguf) | IQ2_XS | 2.03GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ2_XXS.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ2_XXS.gguf) | IQ2_XXS | 1.85GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ1_M.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ1_M.gguf) | IQ1_M | 1.65GB | ✅ Available | 🟢 Yes | 📦 No
+| [RoLlama2-7b-Instruct.IQ1_S.gguf](https://huggingface.co/legraphista/RoLlama2-7b-Instruct-IMat-GGUF/blob/main/RoLlama2-7b-Instruct.IQ1_S.gguf) | IQ1_S | 1.53GB | ✅ Available | 🟢 Yes | 📦 No
+
+
+## Downloading using huggingface-cli
+First, make sure you have hugginface-cli installed:
+```
+pip install -U "huggingface_hub[cli]"
+```
+Then, you can target the specific file you want:
+```
+huggingface-cli download legraphista/RoLlama2-7b-Instruct-IMat-GGUF --include "RoLlama2-7b-Instruct.Q8_0.gguf" --local-dir ./
+```
+If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
+```
+huggingface-cli download legraphista/RoLlama2-7b-Instruct-IMat-GGUF --include "RoLlama2-7b-Instruct.Q8_0/*" --local-dir RoLlama2-7b-Instruct.Q8_0
+# see FAQ for merging GGUF's
+```
+
+## FAQ
+
+### Why is the IMatrix not applied everywhere?
+According to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1993iro/ggufs_quants_can_punch_above_their_weights_now/), it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results). 
+
+### How do I merge a split GGUF?
+1. Make sure you have `gguf-split` available
+    - To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
+    - Download the appropriate zip for your system from the latest release
+    - Unzip the archive and you should be able to find `gguf-split`
+2. Locate your GGUF chunks folder (ex: `RoLlama2-7b-Instruct.Q8_0`)
+3. Run `gguf-split --merge RoLlama2-7b-Instruct.Q8_0/RoLlama2-7b-Instruct.Q8_0-00001-of-XXXXX.gguf RoLlama2-7b-Instruct.Q8_0.gguf`
+    - Make sure to point `gguf-split` to the first chunk of the split.
+
+---
+
+Got a suggestion? Ping me [@legraphista](https://x.com/legraphista)!
--- a/RoLlama2-7b-Instruct.BF16.gguf
+++ b/RoLlama2-7b-Instruct.BF16.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50b16f4c9d6c60e256a1d5ac215975bf9ff2fd524baffea6c322a79332b0d213
+size 13478170560
--- a/RoLlama2-7b-Instruct.FP16.gguf
+++ b/RoLlama2-7b-Instruct.FP16.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:401beffae84d8f2d06a924aaa775f8a44e86921a930a6e9d057437c7a294037e
+size 13478170560
--- a/RoLlama2-7b-Instruct.IQ1_M.gguf
+++ b/RoLlama2-7b-Instruct.IQ1_M.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06794f1dad101f3c416ec027f028a9b4ad27ae4a67fb05ffdc9d72922b261a53
+size 1650987456
--- a/RoLlama2-7b-Instruct.IQ1_S.gguf
+++ b/RoLlama2-7b-Instruct.IQ1_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2db37e76ee0490098a6771cb7fe80d7c7990e78daaec276b3d7bbe56c458d221
+size 1528598976
--- a/RoLlama2-7b-Instruct.IQ2_M.gguf
+++ b/RoLlama2-7b-Instruct.IQ2_M.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:faacf89082f08969cc4858eb8a5d6f8c428d7963715065d5d354ef750fc9dba3
+size 2359769152
--- a/RoLlama2-7b-Instruct.IQ2_S.gguf
+++ b/RoLlama2-7b-Instruct.IQ2_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6257edbcae7a3a77cbd6fc0bd7b1d27877bc8a08c34054f7305dca491bb532b5
+size 2196584512
--- a/RoLlama2-7b-Instruct.IQ2_XS.gguf
+++ b/RoLlama2-7b-Instruct.IQ2_XS.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e7e4bdb078e6d746d87be51724c486f601e1416a250be71675b013b226a07c3
+size 2034930112
--- a/RoLlama2-7b-Instruct.IQ2_XXS.gguf
+++ b/RoLlama2-7b-Instruct.IQ2_XXS.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21066d9f78da99bcf5b867b2bf6368ee7961842d2915a4032f021ed905b8405c
+size 1854968256
--- a/RoLlama2-7b-Instruct.IQ3_M.gguf
+++ b/RoLlama2-7b-Instruct.IQ3_M.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41b9618cff57af8af81007799821e40ee90bcbda1fe4095ed3ad5581458e7929
+size 3114885312
--- a/RoLlama2-7b-Instruct.IQ3_S.gguf
+++ b/RoLlama2-7b-Instruct.IQ3_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7f3ddf832e04dc0c63eaccf93776d62ec0b47e7d605690122a70a9719f8e566
+size 2948325568
--- a/RoLlama2-7b-Instruct.IQ3_XS.gguf
+++ b/RoLlama2-7b-Instruct.IQ3_XS.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2143ff5038e74d3e9208ab2a6bc7e8e5601f656c317133d2689c9ee4e6d790b6
+size 2796544192
--- a/RoLlama2-7b-Instruct.IQ3_XXS.gguf
+++ b/RoLlama2-7b-Instruct.IQ3_XXS.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:442c4f3175af71f388c9b64ed97bb46d905e2d047e9fa8dc9d86c2adba58e4cf
+size 2585409600
--- a/RoLlama2-7b-Instruct.IQ4_NL.gguf
+++ b/RoLlama2-7b-Instruct.IQ4_NL.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:459aaaa0174a8c885217bb80aa1511739acd67e7d82c492e7947d8a3473f1dda
+size 3825830208
--- a/RoLlama2-7b-Instruct.IQ4_XS.gguf
+++ b/RoLlama2-7b-Instruct.IQ4_XS.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27d5d006a2d9c95b2fa4752de5e02bee3150178be10f7d4ac258ea859799c18c
+size 3619358528
--- a/RoLlama2-7b-Instruct.Q2_K.gguf
+++ b/RoLlama2-7b-Instruct.Q2_K.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:99e978d14afc754e6251121763c4970a5f09268fe8d04fe25cd88365daae108d
+size 2532883008
--- a/RoLlama2-7b-Instruct.Q2_K_S.gguf
+++ b/RoLlama2-7b-Instruct.Q2_K_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:184c80fc61393e1c1eb1258ff25bf3028b660a8fcb0b06a8081627487d8b99a5
+size 2319563328
--- a/RoLlama2-7b-Instruct.Q3_K.gguf
+++ b/RoLlama2-7b-Instruct.Q3_K.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41f1d54a28773163ae62b319dbbe2955b5ac514f160ba536aa8cd37156017545
+size 3298025664
--- a/RoLlama2-7b-Instruct.Q3_K_L.gguf
+++ b/RoLlama2-7b-Instruct.Q3_K_L.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0caa445ded352cd2f0f9e9292b270a17ea2e6335fef6054026065fbb7746245a
+size 3597131968
--- a/RoLlama2-7b-Instruct.Q3_K_S.gguf
+++ b/RoLlama2-7b-Instruct.Q3_K_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:688fcf6d5ab951a5cdc34e2dfd8aa57085acc59db1ce5ef9fcc6b9658f4aac1d
+size 2948325568
--- a/RoLlama2-7b-Instruct.Q4_K.gguf
+++ b/RoLlama2-7b-Instruct.Q4_K.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3473ec34972d1befd057e3c63c9413a95fcfcf8709e67df4c40b918a1c78b46f
+size 4081027392
--- a/RoLlama2-7b-Instruct.Q4_K_S.gguf
+++ b/RoLlama2-7b-Instruct.Q4_K_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c5c1d1f568c70147eca6b1d6033aa885b009f4badf4a6e6fd03f34d905ae51a
+size 3856763200
--- a/RoLlama2-7b-Instruct.Q5_K.gguf
+++ b/RoLlama2-7b-Instruct.Q5_K.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b562e506e50ecf3ad2b8e64ed647076b576371096de44c6f41d7d6205e103a16
+size 4783181888
--- a/RoLlama2-7b-Instruct.Q5_K_S.gguf
+++ b/RoLlama2-7b-Instruct.Q5_K_S.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9c0d676f44f2909803f2c19a36c2d9dbca570d3dc9c5956ec3ac36c069ba77e
+size 4651716672
--- a/RoLlama2-7b-Instruct.Q6_K.gguf
+++ b/RoLlama2-7b-Instruct.Q6_K.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5fd5e2bd6443d02fc1f4edd8197e0fc6c0e6a1ff7d08b566425a7fae2666899b
+size 5529221312
--- a/RoLlama2-7b-Instruct.Q8_0.gguf
+++ b/RoLlama2-7b-Instruct.Q8_0.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:821fee39ae1a0cdeceafc96fb73786853c7c24e4deb4efb2028b998202a91045
+size 7161124800
--- a/imatrix.dat
+++ b/imatrix.dat
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8e6698d78462d681aa42e819516a591abd64114414ca1dd25a66a249a625927
+size 4562196
--- a/imatrix.dataset
+++ b/imatrix.dataset
--- a/imatrix.log
+++ b/imatrix.log
@@ -0,0 +1,161 @@
+main: build = 2998 (9588f196)
+main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
+main: seed  = 1716657411
+llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from RoLlama2-7b-Instruct-IMat-GGUF/RoLlama2-7b-Instruct.gguf (version GGUF V3 (latest))
+llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
+llama_model_loader: - kv   0:                       general.architecture str              = llama
+llama_model_loader: - kv   1:                               general.name str              = RoLlama2-7b-Instruct
+llama_model_loader: - kv   2:                          llama.block_count u32              = 32
+llama_model_loader: - kv   3:                       llama.context_length u32              = 4096
+llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096
+llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
+llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 32
+llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 32
+llama_model_loader: - kv   8:                       llama.rope.freq_base f32              = 10000.000000
+llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
+llama_model_loader: - kv  10:                          general.file_type u32              = 0
+llama_model_loader: - kv  11:                           llama.vocab_size u32              = 32004
+llama_model_loader: - kv  12:                 llama.rope.dimension_count u32              = 128
+llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = llama
+llama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = default
+llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32004]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
+llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32004]   = [0.000000, 0.000000, 0.000000, 0.0000...
+llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32004]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
+llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1
+llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2
+llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
+llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
+llama_model_loader: - kv  22:               general.quantization_version u32              = 2
+llama_model_loader: - type  f32:  291 tensors
+llm_load_vocab: special tokens definition check successful ( 263/32004 ).
+llm_load_print_meta: format           = GGUF V3 (latest)
+llm_load_print_meta: arch             = llama
+llm_load_print_meta: vocab type       = SPM
+llm_load_print_meta: n_vocab          = 32004
+llm_load_print_meta: n_merges         = 0
+llm_load_print_meta: n_ctx_train      = 4096
+llm_load_print_meta: n_embd           = 4096
+llm_load_print_meta: n_head           = 32
+llm_load_print_meta: n_head_kv        = 32
+llm_load_print_meta: n_layer          = 32
+llm_load_print_meta: n_rot            = 128
+llm_load_print_meta: n_embd_head_k    = 128
+llm_load_print_meta: n_embd_head_v    = 128
+llm_load_print_meta: n_gqa            = 1
+llm_load_print_meta: n_embd_k_gqa     = 4096
+llm_load_print_meta: n_embd_v_gqa     = 4096
+llm_load_print_meta: f_norm_eps       = 0.0e+00
+llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
+llm_load_print_meta: f_clamp_kqv      = 0.0e+00
+llm_load_print_meta: f_max_alibi_bias = 0.0e+00
+llm_load_print_meta: f_logit_scale    = 0.0e+00
+llm_load_print_meta: n_ff             = 11008
+llm_load_print_meta: n_expert         = 0
+llm_load_print_meta: n_expert_used    = 0
+llm_load_print_meta: causal attn      = 1
+llm_load_print_meta: pooling type     = 0
+llm_load_print_meta: rope type        = 0
+llm_load_print_meta: rope scaling     = linear
+llm_load_print_meta: freq_base_train  = 10000.0
+llm_load_print_meta: freq_scale_train = 1
+llm_load_print_meta: n_yarn_orig_ctx  = 4096
+llm_load_print_meta: rope_finetuned   = unknown
+llm_load_print_meta: ssm_d_conv       = 0
+llm_load_print_meta: ssm_d_inner      = 0
+llm_load_print_meta: ssm_d_state      = 0
+llm_load_print_meta: ssm_dt_rank      = 0
+llm_load_print_meta: model type       = 7B
+llm_load_print_meta: model ftype      = all F32
+llm_load_print_meta: model params     = 6.74 B
+llm_load_print_meta: model size       = 25.10 GiB (32.00 BPW) 
+llm_load_print_meta: general.name     = RoLlama2-7b-Instruct
+llm_load_print_meta: BOS token        = 1 '<s>'
+llm_load_print_meta: EOS token        = 2 '</s>'
+llm_load_print_meta: UNK token        = 0 '<unk>'
+llm_load_print_meta: LF token         = 13 '<0x0A>'
+ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
+ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
+ggml_cuda_init: found 1 CUDA devices:
+  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
+llm_load_tensors: ggml ctx size =    0.30 MiB
+llm_load_tensors: offloading 20 repeating layers to GPU
+llm_load_tensors: offloaded 20/33 layers to GPU
+llm_load_tensors:        CPU buffer size = 25705.14 MiB
+llm_load_tensors:      CUDA0 buffer size = 15440.62 MiB
+...................................................................................................
+llama_new_context_with_model: n_ctx      = 512
+llama_new_context_with_model: n_batch    = 512
+llama_new_context_with_model: n_ubatch   = 512
+llama_new_context_with_model: flash_attn = 0
+llama_new_context_with_model: freq_base  = 10000.0
+llama_new_context_with_model: freq_scale = 1
+llama_kv_cache_init:  CUDA_Host KV buffer size =    96.00 MiB
+llama_kv_cache_init:      CUDA0 KV buffer size =   160.00 MiB
+llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB
+llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB
+llama_new_context_with_model:      CUDA0 compute buffer size =   570.57 MiB
+llama_new_context_with_model:  CUDA_Host compute buffer size =    17.01 MiB
+llama_new_context_with_model: graph nodes  = 1030
+llama_new_context_with_model: graph splits = 136
+
+system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
+compute_imatrix: tokenizing the input ..
+compute_imatrix: tokenization took 134.534 ms
+compute_imatrix: computing over 234 chunks with batch_size 512
+compute_imatrix: 1.07 seconds per pass - ETA 4.15 minutes
+[1]6.7005,[2]4.6847,[3]4.5108,[4]5.0395,[5]5.6345,[6]5.7096,[7]5.1835,[8]5.5952,[9]5.7962,
+save_imatrix: stored collected data after 10 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[10]6.1092,[11]6.1191,[12]5.5838,[13]5.7159,[14]5.6740,[15]6.1152,[16]6.2804,[17]6.6363,[18]6.8269,[19]7.0312,
+save_imatrix: stored collected data after 20 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[20]7.2110,[21]7.2486,[22]7.3765,[23]7.0451,[24]6.8337,[25]6.8506,[26]6.5745,[27]6.4531,[28]6.2121,[29]6.2059,
+save_imatrix: stored collected data after 30 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[30]6.3211,[31]6.4225,[32]6.4934,[33]6.4554,[34]6.4658,[35]6.4658,[36]6.2056,[37]6.0473,[38]5.9893,[39]5.9557,
+save_imatrix: stored collected data after 40 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[40]5.9206,[41]5.8352,[42]5.8871,[43]5.9102,[44]5.9600,[45]6.0277,[46]6.1043,[47]6.1477,[48]6.2849,[49]6.3972,
+save_imatrix: stored collected data after 50 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[50]6.5215,[51]6.6117,[52]6.7180,[53]6.7125,[54]6.6487,[55]6.5960,[56]6.6775,[57]6.7240,[58]6.7473,[59]6.8159,
+save_imatrix: stored collected data after 60 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[60]6.8965,[61]6.9310,[62]6.9934,[63]7.0350,[64]7.0991,[65]7.1198,[66]7.1559,[67]7.1915,[68]7.2275,[69]7.2732,
+save_imatrix: stored collected data after 70 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[70]7.3077,[71]7.3522,[72]7.3969,[73]7.3381,[74]7.2846,[75]7.2409,[76]7.2044,[77]7.2001,[78]7.1674,[79]7.1217,
+save_imatrix: stored collected data after 80 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[80]7.0613,[81]7.0511,[82]7.0086,[83]6.9737,[84]6.9908,[85]7.0142,[86]7.0291,[87]7.0622,[88]7.0617,[89]7.0343,
+save_imatrix: stored collected data after 90 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[90]7.0282,[91]7.0298,[92]7.0163,[93]7.0261,[94]7.0124,[95]7.0157,[96]7.0354,[97]7.0555,[98]7.0258,[99]6.9884,
+save_imatrix: stored collected data after 100 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[100]6.9991,[101]7.0214,[102]7.0149,[103]6.9958,[104]6.9613,[105]6.9460,[106]6.9636,[107]6.9742,[108]6.9615,[109]6.9532,
+save_imatrix: stored collected data after 110 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[110]6.9334,[111]6.9471,[112]6.9669,[113]6.9714,[114]6.9936,[115]6.9930,[116]6.9924,[117]6.9886,[118]7.0000,[119]6.9776,
+save_imatrix: stored collected data after 120 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[120]6.9771,[121]6.9669,[122]6.9392,[123]6.9593,[124]6.9526,[125]6.9559,[126]6.9442,[127]6.9412,[128]6.9540,[129]6.9283,
+save_imatrix: stored collected data after 130 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[130]6.9012,[131]6.8814,[132]6.8839,[133]6.8495,[134]6.8527,[135]6.8238,[136]6.7985,[137]6.7638,[138]6.7283,[139]6.6945,
+save_imatrix: stored collected data after 140 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[140]6.6647,[141]6.6382,[142]6.6132,[143]6.6142,[144]6.6093,[145]6.5929,[146]6.5679,[147]6.5633,[148]6.5580,[149]6.5536,
+save_imatrix: stored collected data after 150 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[150]6.5441,[151]6.5279,[152]6.5201,[153]6.5088,[154]6.4984,[155]6.5112,[156]6.4883,[157]6.4857,[158]6.4982,[159]6.4920,
+save_imatrix: stored collected data after 160 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[160]6.4920,[161]6.5029,[162]6.5026,[163]6.5179,[164]6.5230,[165]6.5400,[166]6.5462,[167]6.5466,[168]6.5516,[169]6.5563,
+save_imatrix: stored collected data after 170 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[170]6.5812,[171]6.5788,[172]6.5882,[173]6.6165,[174]6.6283,[175]6.6511,[176]6.6671,[177]6.6859,[178]6.6988,[179]6.7259,
+save_imatrix: stored collected data after 180 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[180]6.7385,[181]6.7897,[182]6.8050,[183]6.8292,[184]6.8312,[185]6.8364,[186]6.8479,[187]6.8537,[188]6.8407,[189]6.8462,
+save_imatrix: stored collected data after 190 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[190]6.8543,[191]6.8644,[192]6.8716,[193]6.9020,[194]6.8978,[195]6.8769,[196]6.9122,[197]6.9455,[198]6.9734,[199]7.0272,
+save_imatrix: stored collected data after 200 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[200]7.0753,[201]7.0835,[202]7.0873,[203]7.0552,[204]7.0582,[205]7.0700,[206]7.0962,[207]7.0864,[208]7.0836,[209]7.0864,
+save_imatrix: stored collected data after 210 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[210]7.0981,[211]7.1154,[212]7.1139,[213]7.1097,[214]7.1161,[215]7.1363,[216]7.1500,[217]7.1572,[218]7.1493,[219]7.1459,
+save_imatrix: stored collected data after 220 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[220]7.1379,[221]7.1379,[222]7.1357,[223]7.1600,[224]7.1470,[225]7.1489,[226]7.1394,[227]7.1735,[228]7.2152,[229]7.2590,
+save_imatrix: stored collected data after 230 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+[230]7.2989,[231]7.3122,[232]7.2983,[233]7.2814,[234]7.2620,
+save_imatrix: stored collected data after 234 chunks in RoLlama2-7b-Instruct-IMat-GGUF/imatrix.dat
+
+llama_print_timings:        load time =    2800.86 ms
+llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings: prompt eval time =  230095.46 ms / 119808 tokens (    1.92 ms per token,   520.69 tokens per second)
+llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings:       total time =  232816.11 ms / 119809 tokens
+
+Final estimate: PPL = 7.2620 +/- 0.07084