Mungert/Mistral-NeMo-Minitron-8B-Instruct-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Mungert/Mistral-NeMo-Minitron-8B-Instruct-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-13 02:14:18 +08:00
commit c96aed03ae
30 changed files with 381 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
.gitattributes vendored Normal file
View File

@@ -0,0 +1,77 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-f16.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-f16_q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-bf16_q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-f16_q6_k.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-bf16_q6_k.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-f16_q4_k.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-bf16_q4_k.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q2_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q3_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q6_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q2_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q2_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q3_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q3_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q6_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_1.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_0_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q4_1_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_1.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_0_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-q5_1_l.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq2_xs.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq2_xxs.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq2_s.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq2_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq3_xs.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq3_xxs.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq3_s.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq3_m.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq4_xs.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-iq4_nl.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct.imatrix filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-bf16.gguf filter=lfs diff=lfs merge=lfs -text

3
Mistral-NeMo-Minitron-8B-Instruct-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7e869f1aabca13feaa6f85d586cf1aa7b6e9b5ccb967963ba300725843e4b575
size 16836757120

3
Mistral-NeMo-Minitron-8B-Instruct-bf16_q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6e4543e2f909425929ecb0c50e0f22d2eecc825820715f9792e9dea2a407fd80
size 12511381120

3
Mistral-NeMo-Minitron-8B-Instruct-f16_q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:15962ff6b750d20fa8846bb07a9b5781650e8b0fb8cf67c4a431edba8f2cc8c2
size 12511381120

3
Mistral-NeMo-Minitron-8B-Instruct-iq2_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c378a0522c00281045970d7fde06128c05f86ef424787c38f1997af64dad83d7
size 3322644416

3
Mistral-NeMo-Minitron-8B-Instruct-iq2_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:943d51878a1140a1daff91fedce4bf7b3914f411ac8567f8e353da1cdb15b3b0
size 3188950976

3
Mistral-NeMo-Minitron-8B-Instruct-iq2_xs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39fb5c9b8fdfa635309165e3398a74a4aaa3c63fe5611a99cd343da3eec6acdf
size 3096872896

3
Mistral-NeMo-Minitron-8B-Instruct-iq2_xxs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bf14f84677fcdd844b305795d38becefe6bf6d6dd670b62419fa8db5005780ae
size 2888796096

3
Mistral-NeMo-Minitron-8B-Instruct-iq3_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:098c8a76bb670e8aa89e908fbddc27684ab67ace08ee1485efb5620ce6c6c587
size 4136536000

3
Mistral-NeMo-Minitron-8B-Instruct-iq3_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2d5ee156a94e969201721594b1f068b981145761bef55abd867049ccf0be7545
size 4136536000

3
Mistral-NeMo-Minitron-8B-Instruct-iq3_xs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8394e9c1ac6c2ad28598b9b74d0b63c813d9ab33c44ae76494245de0fb511b49
size 3771041728

3
Mistral-NeMo-Minitron-8B-Instruct-iq3_xxs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bd58d90a1afd11319fcc764f6264a0b4f4e58e956b3e525977ccde544533696a
size 3674376128

3
Mistral-NeMo-Minitron-8B-Instruct-iq4_nl.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3282c73d116ed3dc50b3fd01c3e004566552484fb43184e54376abf06865572f
size 4901341120

3
Mistral-NeMo-Minitron-8B-Instruct-iq4_xs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6e911f26bfb0a080d64062f4a3f69685230d0af3844103264d7c6c39f2b56eeb
size 4660430784

3
Mistral-NeMo-Minitron-8B-Instruct-q2_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f205ad04b4869547e900c458aec147237da5ab4ecd31e90a62d4514327568c97
size 3550185408

3
Mistral-NeMo-Minitron-8B-Instruct-q2_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1e8b535aacee313894fb645ff7eb82eedc7ad86ce6a2885ea0bf7f71b64bf3c5
size 3407579072

3
Mistral-NeMo-Minitron-8B-Instruct-q3_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ebc8369e7e09bbc887d4d8795bdb24509a9eb17803a004bac6c25a12493a70c2
size 4389242816

3
Mistral-NeMo-Minitron-8B-Instruct-q3_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:713c41f9a9271003b1a201970c67fdfbe388a895bc5aeff0269a31a185766bf3
size 4246636480

3
Mistral-NeMo-Minitron-8B-Instruct-q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c0387f7eb39856b754cc852432110baf45a142b697d2074c1555238e44be85e
size 4741957568

3
Mistral-NeMo-Minitron-8B-Instruct-q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9f5a94a83e1385dcb31fa91ff703611f4c683f2138d12bbb4379e7f0f024365f
size 5267818432

3
Mistral-NeMo-Minitron-8B-Instruct-q4_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e1d5433d1e33f45fdec96d89892e4d7bde8cc8b591d353bd3a3018b9199d6d27
size 5245896640

3
Mistral-NeMo-Minitron-8B-Instruct-q4_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b94710cbe3135e861d0d0b443423643411ec8cbb378e92490322d8e2f2b61f72
size 5092837312

3
Mistral-NeMo-Minitron-8B-Instruct-q5_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3e9e0cfdd775ad3b2bc79303d1fd29110161da0ee36921db1f750cde35d00752
size 5793679296

3
Mistral-NeMo-Minitron-8B-Instruct-q5_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b0c2f97b9f1a608a013e53517ccb746541810b148fc43ec01f26fe25a9c18bbf
size 6319540160

3
Mistral-NeMo-Minitron-8B-Instruct-q5_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:530e514902c2fad2d20e5941c6ff301f0699435d536136a1d9ef560e2b1b1c4a
size 6090590144

3
Mistral-NeMo-Minitron-8B-Instruct-q5_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:67da97e419700ad36d761cb383a327eb3ea5756089d449e38e048893a171df5e
size 6008702912

3
Mistral-NeMo-Minitron-8B-Instruct-q6_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a6ff1ea3d418bfc6bfccf2ba8dc3b325477c9ca2459b365401e458885a644d88
size 6911133632

3
Mistral-NeMo-Minitron-8B-Instruct-q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:35b422f873c88fcdc0a9305df1e12b2267510c68caf6f5af8960d406b46e61a3
size 8948844160

3
Mistral-NeMo-Minitron-8B-Instruct.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1410fcd8a07c4f5fdebc28c0d2d449aab504520856744cf15bfcf401ad1d2f05
size 5784658

220
README.md Normal file
View File

@@ -0,0 +1,220 @@
---
license: other
license_name: nvidia-open-model-license
license_link: >-
https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
library_name: transformers
base_model:
- nvidia/Mistral-NeMo-Minitron-8B-Base
---
# <span style="color: #7FFF7F;">Mistral-NeMo-Minitron-8B-Instruct GGUF Models</span>
## <span style="color: #7F7FFF;">Model Generation Details</span>
This model was generated using [llama.cpp](https://github.com/ggerganov/llama.cpp) at commit [`e743cddb`](https://github.com/ggerganov/llama.cpp/commit/e743cddb60dc3a8815b9de7dd7d5c491e61b2259).
---
## <span style="color: #7FFF7F;">Quantization Beyond the IMatrix</span>
I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.
In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the `--tensor-type` option in `llama.cpp` to manually "bump" important layers to higher precision. You can see the implementation here:
👉 [Layer bumping with llama.cpp](https://github.com/Mungert69/GGUFModelBuilder/blob/main/model-converter/tensor_list_builder.py)
While this does increase model file size, it significantly improves precision for a given quantization level.
### **I'd love your feedback—have you tried this? How does it perform for you?**
---
<a href="https://readyforquantum.com/huggingface_gguf_selection_guide.html" style="color: #7FFF7F;">
Click here to get info on choosing the right GGUF model format
</a>
---
<!--Begin Original Model Card-->
# Mistral-NeMo-Minitron-8B-Instruct
## Model Overview
Mistral-NeMo-Minitron-8B-Instruct is a model for generating responses for various text-generation tasks including roleplaying, retrieval augmented generation, and function calling. It is a fine-tuned version of [nvidia/Mistral-NeMo-Minitron-8B-Base](https://huggingface.co/nvidia/Mistral-NeMo-Minitron-8B-Base), which was pruned and distilled from [Mistral-NeMo 12B](https://huggingface.co/nvidia/Mistral-NeMo-12B-Base) using [our LLM compression technique](https://arxiv.org/abs/2407.14679). The model was trained using a multi-stage SFT and preference-based alignment technique with [NeMo Aligner](https://github.com/NVIDIA/NeMo-Aligner). For details on the alignment technique, please refer to the [Nemotron-4 340B Technical Report](https://arxiv.org/abs/2406.11704). The model supports a context length of 8,192 tokens.
Try this model on [build.nvidia.com](https://build.nvidia.com/nvidia/mistral-nemo-minitron-8b-8k-instruct).
**Model Developer:** NVIDIA
**Model Dates:** Mistral-NeMo-Minitron-8B-Instruct was trained between August 2024 and September 2024.
## License
[NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf)
## Model Architecture
Mistral-NeMo-Minitron-8B-Instruct uses a model embedding size of 4096, 32 attention heads, MLP intermediate dimension of 11520, with 40 layers in total. Additionally, it uses Grouped-Query Attention (GQA) and Rotary Position Embeddings (RoPE).
**Architecture Type:** Transformer Decoder (Auto-regressive Language Model)
**Network Architecture:** Mistral-NeMo
## Prompt Format:
We recommend using the following prompt template, which was used to fine-tune the model. The model may not perform optimally without it.
```
<extra_id_0>System
{system prompt}
<extra_id_1>User
{prompt}
<extra_id_1>Assistant\n
```
- Note that a newline character `\n` should be added at the end of the prompt.
- We recommend using `<extra_id_1>` as a stop token.
## Usage
```
from transformers import AutoTokenizer, AutoModelForCausalLM
# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("nvidia/Mistral-NeMo-Minitron-8B-Instruct")
model = AutoModelForCausalLM.from_pretrained("nvidia/Mistral-NeMo-Minitron-8B-Instruct")
# Use the prompt template
messages = [
{
"role": "system",
"content": "You are a friendly chatbot who always responds in the style of a pirate",
},
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(tokenized_chat, stop_strings=["<extra_id_1>"], tokenizer=tokenizer)
print(tokenizer.decode(outputs[0]))
```
You can also use `pipeline` but you need to create a tokenizer object and assign it to the pipeline manually.
```
from transformers import AutoTokenizer
from transformers import pipeline
tokenizer = AutoTokenizer.from_pretrained("nvidia/Mistral-NeMo-Minitron-8B-Instruct")
messages = [
{"role": "user", "content": "Who are you?"},
]
pipe = pipeline("text-generation", model="nvidia/Mistral-NeMo-Minitron-8B-Instruct")
pipe(messages, max_new_tokens=64, stop_strings=["<extra_id_1>"], tokenizer=tokenizer)
```
## Evaluation Results
| Category | Benchmark | # Shots | Mistral-NeMo-Minitron-8B-Instruct |
|:----------------------|:----------------------|--------:|----------------------------------:|
| General | MMLU | 5 | 70.4 |
| | MT Bench (GPT4-Turbo) | 0 | 7.86 |
| Math | GMS8K | 0 | 87.1 |
| Reasoning | GPQA | 0 | 31.5 |
| Code | HumanEval | 0 | 71.3 |
| | MBPP | 0 | 72.5 |
| Instruction Following | IFEval | 0 | 84.4 |
| Tool Use | BFCL v2 Live | 0 | 67.6 |
## AI Safety Efforts
The Mistral-NeMo-Minitron-8B-Instruct model underwent AI safety evaluation including adversarial testing via three distinct methods:
- [Garak](https://github.com/leondz/garak), is an automated LLM vulnerability scanner that probes for common weaknesses, including prompt injection and data leakage.
- [AEGIS](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-1.0), is a content safety evaluation dataset and LLM based content safety classifier model, that adheres to a broad taxonomy of 13 categories of critical risks in human-LLM interactions.
- Human Content Red Teaming leveraging human interaction and evaluation of the models' responses.
## Limitations
The model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. This issue could be exacerbated without the use of the recommended prompt template. This issue could be exacerbated without the use of the recommended prompt template. If you are going to use this model in an agentic workflow, validate that the imported packages are from a trusted source to ensure end-to-end security.
## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the [Model Card++](https://build.nvidia.com/nvidia/mistral-nemo-minitron-8b-8k-instruct/modelcard). Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
<!--End Original Model Card-->
---
# <span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>
Help me test my **AI-Powered Quantum Network Monitor Assistant** with **quantum-ready security checks**:
👉 [Quantum Network Monitor](https://readyforquantum.com/?assistant=open&utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme)
The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : [Source Code Quantum Network Monitor](https://github.com/Mungert69). You will also find the code I use to quantize the models if you want to do it yourself [GGUFModelBuilder](https://github.com/Mungert69/GGUFModelBuilder)
💬 **How to test**:
Choose an **AI assistant type**:
- `TurboLLM` (GPT-4.1-mini)
- `HugLLM` (Hugginface Open-source models)
- `TestLLM` (Experimental CPU-only)
### **What I’m Testing**
I’m pushing the limits of **small open-source models for AI network monitoring**, specifically:
- **Function calling** against live network services
- **How small can a model go** while still handling:
- Automated **Nmap security scans**
- **Quantum-readiness checks**
- **Network Monitoring tasks**
🟡 **TestLLM** – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):
- ✅ **Zero-configuration setup**
- ⏳ 30s load time (slow inference but **no API costs**) . No token limited as the cost is low.
- 🔧 **Help wanted!** If you’re into **edge-device AI**, let’s collaborate!
### **Other Assistants**
🟢 **TurboLLM** – Uses **gpt-4.1-mini** :
- **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
- **Create custom cmd processors to run .net code on Quantum Network Monitor Agents**
- **Real-time network diagnostics and monitoring**
- **Security Audits**
- **Penetration testing** (Nmap/Metasploit)
🔵 **HugLLM** – Latest Open-source models:
- 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.
### 💡 **Example commands you could test**:
1. `"Give me info on my websites SSL certificate"`
2. `"Check if my server is using quantum safe encyption for communication"`
3. `"Run a comprehensive security audit on my server"`
4. '"Create a cmd processor to .. (what ever you want)" Note you need to install a [Quantum Network Monitor Agent](https://readyforquantum.com/Download/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) to run the .net code on. This is a very flexible and powerful feature. Use with caution!
### Final Word
I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is [open source](https://github.com/Mungert69). Feel free to use whatever you find helpful.
If you appreciate the work, please consider [buying me a coffee](https://www.buymeacoffee.com/mahadeva) ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
I'm also open to job opportunities or sponsorship.
Thank you! 😊