Mungert/Llama-3.1-Minitron-4B-Width-Base-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Mungert/Llama-3.1-Minitron-4B-Width-Base-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-09 09:29:17 +08:00
commit 467d4a4c94
30 changed files with 416 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
.gitattributes vendored Normal file
View File

@@ -0,0 +1,77 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-f16_q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-bf16_q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-f16_q6_k.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-bf16_q6_k.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-f16_q4_k.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-bf16_q4_k.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q2_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q3_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q6_k_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q2_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q2_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q3_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q3_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_k_s.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q6_k_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_1.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_0_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q4_1_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_1.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_0_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-q5_1_l.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq2_xs.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq2_xxs.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq2_s.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq2_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq3_xs.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq3_xxs.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq3_s.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq3_m.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq4_xs.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-iq4_nl.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base.imatrix filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-bf16.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-Minitron-4B-Width-Base-f16.gguf filter=lfs diff=lfs merge=lfs -text

3
Llama-3.1-Minitron-4B-Width-Base-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f927103eb2ee4cc2e2a0e54473590e6024919bca021da2cb1c76798a35b24199
size 9033729088

3
Llama-3.1-Minitron-4B-Width-Base-bf16_q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ab78485383a62e9e8d97e2af41152529e382fcc588ceef57f8cd5a29bfce10bc
size 6863176768

3
Llama-3.1-Minitron-4B-Width-Base-f16_q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c2085883497c1b42ee6a8b8eca1a25ee9d5b140135a0c375e8344409a576896
size 6863176768

3
Llama-3.1-Minitron-4B-Width-Base-iq2_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9bdaa0772d70473a3b5a4b3eb0759532067b8aa478e26bbf525449ff00167a20
size 1862108544

3
Llama-3.1-Minitron-4B-Width-Base-iq2_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:25c686fd8d576e5bd46b8189f1cb572cb0011851f1ee778475e836b6fb8ac80c
size 1796638080

3
Llama-3.1-Minitron-4B-Width-Base-iq2_xs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bb78add21dc158d46ee6d269e252ef0474a8235b4b6f09be5cb8243df31577ca
size 1751565696

3
Llama-3.1-Minitron-4B-Width-Base-iq2_xxs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f216f2aa0e813db6695758ccf53a0ca2d46f1006e59a4e79b12d1e066a7f8142
size 1644119424

3
Llama-3.1-Minitron-4B-Width-Base-iq3_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:897badf169d6b7da7bd960e8e6fe7c8e288bd7b1a90ea059e46605b47d358b8f
size 2271102336

3
Llama-3.1-Minitron-4B-Width-Base-iq3_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54ef350bf1f9b2e91d2c8ce9e16958d219abcf1f5dc506409b2ce4c9324f8768
size 2271102336

3
Llama-3.1-Minitron-4B-Width-Base-iq3_xs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:927da0f14261699dde1661caa41fac5563b906feb9c333104d9c749340eaf464
size 2089043328

3
Llama-3.1-Minitron-4B-Width-Base-iq3_xxs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:13320213c40658768bd85e22e76efc57acad17d25605a40959e834f2101ac38f
size 2036352384

3
Llama-3.1-Minitron-4B-Width-Base-iq4_nl.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:da02be10772b8f99ffbb357e6b30a47b9ce32f9397ffa3de8a02f98f8d0aebab
size 2661105024

3
Llama-3.1-Minitron-4B-Width-Base-iq4_xs.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ee489499578c91e5612b5dd7109cbd77a03fa4f94f8cd9b247de6c4d6f7c5d3b
size 2535546240

3
Llama-3.1-Minitron-4B-Width-Base-q2_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bfb7030191df818ffd9e3faf82c347654311df487313984f20a0837f685662d7
size 2006136192

3
Llama-3.1-Minitron-4B-Width-Base-q2_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3ded1a083a136a3aadc46045c991bcf82a60798ab42f8f01cb48be6c2d1c4b94
size 1901479296

3
Llama-3.1-Minitron-4B-Width-Base-q3_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:52499fd864cedf2756862e6a475c924f5fa5c040113790847b7690b45c096f1c
size 2433168768

3
Llama-3.1-Minitron-4B-Width-Base-q3_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ff0a3e1686ff800b05a0b416b93cefcc18b6ef9d4b475160792b9ee16059ce87
size 2328511872

3
Llama-3.1-Minitron-4B-Width-Base-q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:15839b0d7e3eb1e058b15ad4ac0a666a9d07b1116eb48f35ebcd0d1188101f17
size 2546943360

3
Llama-3.1-Minitron-4B-Width-Base-q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0265e6e3ae22c0f3eccc59163e5860b7ec8419e035b317455eaa155119f448a6
size 2828977536

3
Llama-3.1-Minitron-4B-Width-Base-q4_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54ddbcc1abfe59b0ddad18dcfba3c7789c7328b8ad4e5ecf38dff35f298b8671
size 2859021696

3
Llama-3.1-Minitron-4B-Width-Base-q4_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:65e2639d445ec93a2da9e633862291ebaaf4c01156b2b92a51bb16081dfa5867
size 2794141056

3
Llama-3.1-Minitron-4B-Width-Base-q5_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c10cc5b9a8ab913e0966beba14673acee52555d9bb4672c974757abac87ccd3f
size 3111011712

3
Llama-3.1-Minitron-4B-Width-Base-q5_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:981353e71dd82fdb0057701a78c4a3b123afc02b303517eea607eaf20ee5f7f8
size 3393045888

3
Llama-3.1-Minitron-4B-Width-Base-q5_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ce0f35cbac689eb49c056f92b4beeee9ff1ca5c9412d99a945e9de5ca215a598
size 3295884672

3
Llama-3.1-Minitron-4B-Width-Base-q5_k_s.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:17b0033ae29e61f88cd0bf86f5d625de6b44bee510f8b09bf1fa9f8396a868e3
size 3262461312

3
Llama-3.1-Minitron-4B-Width-Base-q6_k_m.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6c870b734609d810fb2338baa9860b0ab5db8c0e3f9bca9e4a7d75642f03d5e3
size 3710334336

3
Llama-3.1-Minitron-4B-Width-Base-q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c1eb9e6b0b7ba6cf2ec17befc84ffb6b37ae026e41f740170caff63e86f62cbf
size 4803216448

3
Llama-3.1-Minitron-4B-Width-Base.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f1cc5cfb9133c6ff4a4b6ad31fdd3195950d34f18f7a896e5ee771bcf71a3af0
size 3677450

255
README.md Normal file
View File

@@ -0,0 +1,255 @@
---
license: other
license_name: nvidia-open-model-license
license_link: >-
https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- nvidia
- llama-3
- pytorch
---
# <span style="color: #7FFF7F;">Llama-3.1-Minitron-4B-Width-Base GGUF Models</span>
## <span style="color: #7F7FFF;">Model Generation Details</span>
This model was generated using [llama.cpp](https://github.com/ggerganov/llama.cpp) at commit [`b9c3eefd`](https://github.com/ggerganov/llama.cpp/commit/b9c3eefde1b67104bd993485ff38dd62abe9d70c).
---
## <span style="color: #7FFF7F;">Quantization Beyond the IMatrix</span>
I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.
In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the `--tensor-type` option in `llama.cpp` to manually "bump" important layers to higher precision. You can see the implementation here:
👉 [Layer bumping with llama.cpp](https://github.com/Mungert69/GGUFModelBuilder/blob/main/model-converter/tensor_list_builder.py)
While this does increase model file size, it significantly improves precision for a given quantization level.
### **I'd love your feedback—have you tried this? How does it perform for you?**
---
<a href="https://readyforquantum.com/huggingface_gguf_selection_guide.html" style="color: #7FFF7F;">
Click here to get info on choosing the right GGUF model format
</a>
---
<!--Begin Original Model Card-->
# Llama-3.1-Minitron-4B-Width-Base
## Model Overview
Llama-3.1-Minitron-4B-Width-Base is a base text-to-text model that can be adopted for a variety of natural language generation tasks.
It is obtained by pruning Llama-3.1-8B; specifically, we prune model embedding size and MLP intermediate dimension.
Following pruning, we perform continued training with distillation using 94 billion tokens to arrive at the final model; we use the continuous pre-training data corpus used in Nemotron-4 15B for this purpose. Please refer to our [technical report](https://arxiv.org/abs/2408.11796) for more details.
This model is ready for commercial use.
**Model Developer:** NVIDIA
**Model Dates:** Llama-3.1-Minitron-4B-Width-Base was trained between July 29, 2024 and Aug 3, 2024.
## License
This model is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
## Model Architecture
Llama-3.1-Minitron-4B-Width-Base uses a model embedding size of 3072, 32 attention heads, MLP intermediate dimension of 9216, with 32 layers in total. Additionally, it uses Grouped-Query Attention (GQA) and Rotary Position Embeddings (RoPE).
**Architecture Type:** Transformer Decoder (Auto-Regressive Language Model)
**Network Architecture:** Llama-3.1
**Input Type(s):** Text
**Input Format(s):** String
**Input Parameters:** None
**Other Properties Related to Input:** Works well within 8k characters or less.
**Output Type(s):** Text
**Output Format:** String
**Output Parameters:** 1D
**Other Properties Related to Output:** None
## Usage
Support for this model will be added in the upcoming `transformers` release. In the meantime, please install the library from source:
```
pip install git+https://github.com/huggingface/transformers
```
We can now run inference on this model:
```python
import torch
from transformers import AutoTokenizer, LlamaForCausalLM
# Load the tokenizer and model
model_path = "nvidia/Llama-3.1-Minitron-4B-Width-Base"
tokenizer = AutoTokenizer.from_pretrained(model_path)
device = 'cuda'
dtype = torch.bfloat16
model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=dtype, device_map=device)
# Prepare the input text
prompt = 'Complete the paragraph: our solar system is'
inputs = tokenizer.encode(prompt, return_tensors='pt').to(model.device)
# Generate the output
outputs = model.generate(inputs, max_length=20)
# Decode and print the output
output_text = tokenizer.decode(outputs[0])
print(output_text)
```
## Software Integration
**Runtime Engine(s):**
* NeMo 24.05
**Supported Hardware Microarchitecture Compatibility:** <br>
* NVIDIA Ampere
* NVIDIA Blackwell
* NVIDIA Hopper
* NVIDIA Lovelace
**[Preferred/Supported] Operating System(s):** <br>
* Linux
## Dataset & Training
**Data Collection Method by Dataset:** Automated
**Labeling Method by Dataset:** Not Applicable
**Properties:**
The training corpus for Llama-3.1-Minitron-4B-Width-Base consists of English and multilingual text, as well as code. Our sources cover a variety of document types such as: webpages, dialogue, articles, and other written materials. The corpus spans domains including legal, math, science, finance, and more. In our continued training set, we introduce a small portion of question-answering, and alignment style data to improve model performance.
**Data Freshness:** The pretraining data has a cutoff of June 2023.
## Evaluation Results
### Overview
_5-shot performance._ Language Understanding evaluated using [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300):
| Average |
| :---- |
| 60.5 |
_Zero-shot performance._ Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) with additions:
| HellaSwag | Winogrande | GSM8K| ARC-Challenge | XLSum |
| :---- | :---- | :---- | :---- | :---- |
| 76.1 | 73.5 | 41.2 | 55.6 | 28.7
_Code generation performance._ Evaluated using [MBPP](https://github.com/google-research/google-research/tree/master/mbpp):
| Score |
| :---- |
| 32.0 |
## Inference
**Engine:** TensorRT-LLM
**Test Hardware:** NVIDIA A100
**DType:** BFloat16
## Limitations
The model was trained on data that contains toxic language, unsafe content, and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.
## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
## References
* [Compact Language Models via Pruning and Knowledge Distillation](https://arxiv.org/abs/2407.14679)
* [LLM Pruning and Distillation in Practice: The Minitron Approach](https://arxiv.org/abs/2408.11796)
<!--End Original Model Card-->
---
# <span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>
Help me test my **AI-Powered Quantum Network Monitor Assistant** with **quantum-ready security checks**:
👉 [Quantum Network Monitor](https://readyforquantum.com/?assistant=open&utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme)
The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : [Source Code Quantum Network Monitor](https://github.com/Mungert69). You will also find the code I use to quantize the models if you want to do it yourself [GGUFModelBuilder](https://github.com/Mungert69/GGUFModelBuilder)
💬 **How to test**:
Choose an **AI assistant type**:
- `TurboLLM` (GPT-4.1-mini)
- `HugLLM` (Hugginface Open-source models)
- `TestLLM` (Experimental CPU-only)
### **What I’m Testing**
I’m pushing the limits of **small open-source models for AI network monitoring**, specifically:
- **Function calling** against live network services
- **How small can a model go** while still handling:
- Automated **Nmap security scans**
- **Quantum-readiness checks**
- **Network Monitoring tasks**
🟡 **TestLLM** – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):
- ✅ **Zero-configuration setup**
- ⏳ 30s load time (slow inference but **no API costs**) . No token limited as the cost is low.
- 🔧 **Help wanted!** If you’re into **edge-device AI**, let’s collaborate!
### **Other Assistants**
🟢 **TurboLLM** – Uses **gpt-4.1-mini** :
- **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
- **Create custom cmd processors to run .net code on Quantum Network Monitor Agents**
- **Real-time network diagnostics and monitoring**
- **Security Audits**
- **Penetration testing** (Nmap/Metasploit)
🔵 **HugLLM** – Latest Open-source models:
- 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.
### 💡 **Example commands you could test**:
1. `"Give me info on my websites SSL certificate"`
2. `"Check if my server is using quantum safe encyption for communication"`
3. `"Run a comprehensive security audit on my server"`
4. '"Create a cmd processor to .. (what ever you want)" Note you need to install a [Quantum Network Monitor Agent](https://readyforquantum.com/Download/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) to run the .net code on. This is a very flexible and powerful feature. Use with caution!
### Final Word
I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is [open source](https://github.com/Mungert69). Feel free to use whatever you find helpful.
If you appreciate the work, please consider [buying me a coffee](https://www.buymeacoffee.com/mahadeva) ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
I'm also open to job opportunities or sponsorship.
Thank you! 😊