commit b9bdeb1036d2e6a33be157190f65a40a21d5c8f6 Author: ModelHub XC Date: Wed Sep 9 09:52:16 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..f15b49c --- /dev/null +++ b/.gitattributes @@ -0,0 +1,36 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +*.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/ModelfileOllama b/ModelfileOllama new file mode 100644 index 0000000..b8c052d --- /dev/null +++ b/ModelfileOllama @@ -0,0 +1,22 @@ +FROM NeuralNet/openchat-3.6 + +# sets the temperature to 1 [higher is more creative, lower is more coherent] +PARAMETER temperature 0.5 + +# sets the context window size to 8192, this controls how many tokens the LLM can use as context to generate the next token +PARAMETER num_ctx 8192 + +# tokens to generate set to 4096 (max) +PARAMETER num_predict 4096 + +# set system +SYSTEM "You are an AI assistant created by NeuralNet, a company specialized in AI solutions, your answer are clear and consice" + +# template OpenChat3.6 +TEMPLATE "{{ if .System }}<|begin_of_text|><|start_header_id|>System<|end_header_id|> + +{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>GPT4 Correct User<|end_header_id|> + +{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>GPT4 Correct Assistant<|end_header_id|> + +{{ .Response }}<|eot_id|>" diff --git a/README.md b/README.md new file mode 100644 index 0000000..426c9bb --- /dev/null +++ b/README.md @@ -0,0 +1,184 @@ +--- +license: llama3 +base_model: openchat/openchat-3.6-8b-20240522 +tags: + - openchat + - llama3 + - C-RLFT +library_name: transformers +pipeline_tag: text-generation +quantized_by: NeuralNet-Hub +--- + +
+ + + +
+ +NeuralNet is a pioneering AI solutions provider that empowers businesses to harness the power of artificial intelligence + + +## 🌟 OpenChat-3.6-8b-20240522 llama.cpp quantization by NeuralNet 🧠🤖 + +All the models have been quantized following the instructions provided by [`llama.cpp`](https://github.com/ggerganov/llama.cpp/blob/master/README.md#prepare-and-quantize). This is: +``` +# obtain the official LLaMA model weights and place them in ./models +ls ./models +llama-2-7b tokenizer_checklist.chk tokenizer.model +# [Optional] for models using BPE tokenizers +ls ./models + vocab.json +# [Optional] for PyTorch .bin models like Mistral-7B +ls ./models + + +# install Python dependencies +python3 -m pip install -r requirements.txt + +# convert the model to ggml FP16 format +python3 convert-hf-to-gguf.py models/mymodel/ + +# quantize the model to 4-bits (using Q4_K_M method) +./llama-quantize ./models/mymodel/ggml-model-f16.gguf ./models/mymodel/ggml-model-Q4_K_M.gguf Q4_K_M + +# update the gguf filetype to current version if older version is now unsupported +./llama-quantize ./models/mymodel/ggml-model-Q4_K_M.gguf ./models/mymodel/ggml-model-Q4_K_M-v2.gguf COPY +``` + +Original model: https://huggingface.co/openchat/openchat-3.6-8b-20240522 + +## Prompt format 📝 + +### Original Format: +``` +<|begin_of_text|><|start_header_id|>System<|end_header_id|> + +{system}<|eot_id|><|start_header_id|>GPT4 Correct User<|end_header_id|> + +{user}<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|> +``` + +### Ollama Template: +``` +{{ if .System }}<|begin_of_text|><|start_header_id|>System<|end_header_id|> + +{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>GPT4 Correct User<|end_header_id|> + +{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>GPT4 Correct Assistant<|end_header_id|> + +{{ .Response }}<|eot_id|> +``` + +## Summary models 📋 + +| Filename | Quant type | File Size | Description | +| -------- | ---------- | --------- | ----------- | +| [openchat-3.6-8b-20240522-fp16.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-fp16.gguf) | fp16 | 16.06GB | Half precision, no quantization applied | +| [openchat-3.6-8b-20240522-q8_0.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q8_0.gguf) | q8_0 | 8.54GB | Extremely high quality, generally unneeded but max available quant. | +| [openchat-3.6-8b-20240522-q6_K.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q6_K.gguf) | q6_K | 6.59GB | Very high quality, near perfect, *recommended*. | +| [openchat-3.6-8b-20240522-q5_1.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q5_1.gguf) | q5_1 | 6.06GB | High quality, *recommended*. | +| [openchat-3.6-8b-20240522-q5_K_M.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q5_K_M.gguf) | q5_K_M | 5.73GB | High quality, *recommended*. | +| [openchat-3.6-8b-20240522-q5_K_S.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q5_K_S.gguf) | q5_K_S | 5.59GB | High quality, *recommended*. | +| [openchat-3.6-8b-20240522-q5_K_S.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q5_0.gguf) | q5_0 | 5.59GB | High quality, *recommended*. | +| [openchat-3.6-8b-20240522-q4_K_M.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q4_1.gguf) | q4_1 | 4.92GB | Good quality, *recommended*. | +| [openchat-3.6-8b-20240522-q4_K_M.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q4_K_M.gguf) | q4_K_M | 4.92GB | Good quality, uses about 4.83 bits per weight, *recommended*. | +| [openchat-3.6-8b-20240522-q4_K_S.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q4_K_S.gguf) | q4_K_S | 4.69GB | Slightly lower quality with more space savings, *recommended*. | +| [openchat-3.6-8b-20240522-q4_0.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q4_0.gguf) | q4_0 | 4.66GB | Slightly lower quality with more space savings, *recommended*. | +| [openchat-3.6-8b-20240522-q3_K_L.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q3_K_L.gguf) | q3_K_L | 4.32GB | Lower quality but usable, good for low RAM availability. | +| [openchat-3.6-8b-20240522-q3_K_M.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q3_K_M.gguf) | q3_K_M | 4.01GB | Even lower quality. | +| [openchat-3.6-8b-20240522-q3_K_S.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q3_K_S.gguf) | q3_K_S | 3.66GB | Low quality, not recommended. | +| [openchat-3.6-8b-20240522-q2_K.gguf](https://huggingface.co/NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF/blob/main/openchat-3.6-8b-20240522-q2_K.gguf) | q2_K | 3.17GB | Very low quality but surprisingly usable. | + +## Usage with Ollama 🦙 + +### Direct from Ollama +``` +ollama run NeuralNet/openchat-3.6-8b-20240522 +``` + +### Create your own template +Create a text plain file named `Modelfile` (no extension needed) +``` +FROM NeuralNet/openchat-3.6 + +# sets the temperature to 1 [higher is more creative, lower is more coherent] +PARAMETER temperature 0.5 + +# sets the context window size to 8192, this controls how many tokens the LLM can use as context to generate the next token +PARAMETER num_ctx 8192 + +# tokens to generate set to 4096 (max) +PARAMETER num_predict 4096 + +# set system +SYSTEM "You are an AI assistant created by NeuralNet, your answer are clear and consice" + +# template OpenChat3.6 +TEMPLATE "{{ if .System }}<|begin_of_text|><|start_header_id|>System<|end_header_id|> + +{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>GPT4 Correct User<|end_header_id|> + +{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>GPT4 Correct Assistant<|end_header_id|> + +{{ .Response }}<|eot_id|>" +``` +Then, after previously install ollama, just run: +``` +ollama create openchat-3.6-8b-20240522 -f openchat-3.6-8b-20240522 +``` + + +## Download Models Using huggingface-cli 🤗 + +### Installation of `huggingface_hub[cli]` +Ensure you have the necessary CLI tool installed by running: +```bash +pip install -U "huggingface_hub[cli]" +``` + +### Downloading Specific Model Files +To download a specific model file, use the following command: +```bash +huggingface-cli download NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF --include "openchat-3.6-8b-20240522-Q4_K_M.gguf" --local-dir ./ +``` +This command downloads the specified model file and places it in the current directory (./). + +### Downloading Large Models Split into Multiple Files +For models exceeding 50GB, which are typically split into multiple files for easier download and management: +```bash +huggingface-cli download NeuralNet-Hub/openchat-3.6-8b-20240522-GGUF --include "openchat-3.6-8b-20240522-Q8_0.gguf/*" --local-dir openchat-3.6-8b-20240522-Q8_0 +``` +This command downloads all files in the specified directory and places them into the chosen local folder (openchat-3.6-8b-20240522-Q8_0). You can choose to download everything in place or specify a new location for the downloaded files. + +## Which File Should I Choose? 📈 + +A comprehensive analysis with performance charts is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9). + +### Assessing System Capabilities +1. **Determine Your Model Size**: Start by checking the amount of RAM and VRAM available in your system. This will help you decide the largest possible model you can run. +2. **Optimizing for Speed**: + - **GPU Utilization**: To run your model as quickly as possible, aim to fit the entire model into your GPU's VRAM. Pick a version that’s 1-2GB smaller than the total VRAM. +3. **Maximizing Quality**: + - **Combined Memory**: For the highest possible quality, sum your system RAM and GPU's VRAM. Then choose a model that's 1-2GB smaller than this combined total. + +### Deciding Between 'I-Quant' and 'K-Quant' +1. **Simplicity**: + - **K-Quant**: If you prefer a straightforward approach, select a K-quant model. These are labeled as 'QX_K_X', such as Q5_K_M. +2. **Advanced Configuration**: + - **Feature Chart**: For a more nuanced choice, refer to the [llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix). + - **I-Quant Models**: Best suited for configurations below Q4 and for systems running cuBLAS (Nvidia) or rocBLAS (AMD). These are labeled 'IQX_X', such as IQ3_M, and offer better performance for their size. + - **Compatibility Considerations**: + - **I-Quant Models**: While usable on CPU and Apple Metal, they perform slower compared to their K-quant counterparts. The choice between speed and performance becomes a significant tradeoff. + - **AMD Cards**: Verify if you are using the rocBLAS build or the Vulkan build. I-quants are not compatible with Vulkan. + - **Current Support**: At the time of writing, LM Studio offers a preview with ROCm support, and other inference engines provide specific ROCm builds. + +By following these guidelines, you can make an informed decision on which file best suits your system and performance needs. + +## Contact us 🌐 + +NeuralNet is a pioneering AI solutions provider that empowers businesses to harness the power of artificial intelligence + +Website: https://neuralnet.solutions +Email: info[at]neuralnet.solutions + diff --git a/ggml-openchat-3.6-8b-20240522-fp16.gguf b/ggml-openchat-3.6-8b-20240522-fp16.gguf new file mode 100644 index 0000000..7d244f6 --- /dev/null +++ b/ggml-openchat-3.6-8b-20240522-fp16.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:972a2b22563d804a86aabc5e88b42174b7e169a35d60ba983ece0cd8a927d263 +size 16068891136 diff --git a/openchat-3.6-8b-20240522-q2_K.gguf b/openchat-3.6-8b-20240522-q2_K.gguf new file mode 100644 index 0000000..0c96fec --- /dev/null +++ b/openchat-3.6-8b-20240522-q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a4001d84b0d4c3695d94fd25f110ebf8e7fad9cfab8427b4b89b0264d31c660a +size 3179131392 diff --git a/openchat-3.6-8b-20240522-q3_K_L.gguf b/openchat-3.6-8b-20240522-q3_K_L.gguf new file mode 100644 index 0000000..b58acfe --- /dev/null +++ b/openchat-3.6-8b-20240522-q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:23a572a44ea5f2883c8ac39dff2cad94c190970d8fee38a514ebd1613d1b9f27 +size 4321956352 diff --git a/openchat-3.6-8b-20240522-q3_K_M.gguf b/openchat-3.6-8b-20240522-q3_K_M.gguf new file mode 100644 index 0000000..0179915 --- /dev/null +++ b/openchat-3.6-8b-20240522-q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c72258a6fa669c9998def587e99ed02c344a583f1c7536e9a70299613bb20661 +size 4018917888 diff --git a/openchat-3.6-8b-20240522-q3_K_S.gguf b/openchat-3.6-8b-20240522-q3_K_S.gguf new file mode 100644 index 0000000..2b535b4 --- /dev/null +++ b/openchat-3.6-8b-20240522-q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5536f578d548bb276f3080c0bba9c60785b50da4e75b32041b73473e617a4dd8 +size 3664499200 diff --git a/openchat-3.6-8b-20240522-q4_0.gguf b/openchat-3.6-8b-20240522-q4_0.gguf new file mode 100644 index 0000000..127fbaf --- /dev/null +++ b/openchat-3.6-8b-20240522-q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7812e862ab10037bfc3dd8253271b7d4429572faee4d1aec8aa6826ae7b470ec +size 4661211648 diff --git a/openchat-3.6-8b-20240522-q4_1.gguf b/openchat-3.6-8b-20240522-q4_1.gguf new file mode 100644 index 0000000..e824487 --- /dev/null +++ b/openchat-3.6-8b-20240522-q4_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f4d4582dff17f4cc46fe8ce93910c24d3ef9afa8d90d2ea2719ff99a90d309de +size 5130252800 diff --git a/openchat-3.6-8b-20240522-q4_K_M.gguf b/openchat-3.6-8b-20240522-q4_K_M.gguf new file mode 100644 index 0000000..71b9102 --- /dev/null +++ b/openchat-3.6-8b-20240522-q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:097dca9eae0f44b17859a8aa75cb16cae63bfe981da141aa8ec166f9a801ca73 +size 4920734208 diff --git a/openchat-3.6-8b-20240522-q4_K_S.gguf b/openchat-3.6-8b-20240522-q4_K_S.gguf new file mode 100644 index 0000000..9a133f9 --- /dev/null +++ b/openchat-3.6-8b-20240522-q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6ca87f5c94b9b5d4c07303bf8b31a02c1a573952618d7088fce1f2326aaf01ea +size 4692668928 diff --git a/openchat-3.6-8b-20240522-q5_0.gguf b/openchat-3.6-8b-20240522-q5_0.gguf new file mode 100644 index 0000000..fa1e80c --- /dev/null +++ b/openchat-3.6-8b-20240522-q5_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3a7314cfb43fe1190a64fa485d6599db13bbecc6c492015f22012efb4f4a7a67 +size 5599293952 diff --git a/openchat-3.6-8b-20240522-q5_1.gguf b/openchat-3.6-8b-20240522-q5_1.gguf new file mode 100644 index 0000000..5d4b8b0 --- /dev/null +++ b/openchat-3.6-8b-20240522-q5_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:08843f08cca1565794258a5833670de7eb7f87a300f92fd0da17186779bd0a1d +size 6068335104 diff --git a/openchat-3.6-8b-20240522-q5_K_M.gguf b/openchat-3.6-8b-20240522-q5_K_M.gguf new file mode 100644 index 0000000..ee1ff42 --- /dev/null +++ b/openchat-3.6-8b-20240522-q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8c0706288d7a8085698eef63455a5c06bd803dc1e2befce6a81a210058b2ac5e +size 5732987392 diff --git a/openchat-3.6-8b-20240522-q5_K_S.gguf b/openchat-3.6-8b-20240522-q5_K_S.gguf new file mode 100644 index 0000000..ceef648 --- /dev/null +++ b/openchat-3.6-8b-20240522-q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:282eaaeacdc1b71785ee594832099c8a49f78362c1d3a8bf8539bda24dec197c +size 5599293952 diff --git a/openchat-3.6-8b-20240522-q6_K.gguf b/openchat-3.6-8b-20240522-q6_K.gguf new file mode 100644 index 0000000..17c91d0 --- /dev/null +++ b/openchat-3.6-8b-20240522-q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:83c4ab6e1880c6cdf0ee9a739f0f05ef7436c62ba47964324e621926af84f573 +size 6596006400 diff --git a/openchat-3.6-8b-20240522-q8_0.gguf b/openchat-3.6-8b-20240522-q8_0.gguf new file mode 100644 index 0000000..0d6f6b1 --- /dev/null +++ b/openchat-3.6-8b-20240522-q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ba8b98f426fa8ef262ec9c2127080fbe22201fd7fa25fb907a7067761025528b +size 8540770816