164 lines
7.5 KiB
Markdown
164 lines
7.5 KiB
Markdown
---
|
||
license: gemma
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- gemma
|
||
---
|
||
|
||
# <span style="color: #7FFF7F;">Gemma-3 1B Instruct GGUF Models</span>
|
||
|
||
|
||
**Note llama-quantize was not able to fully quantize the ggufs for k quants as the tensor dimensions of some weights where not divisible by 256. fallback quants where used.**
|
||
|
||
## **Choosing the Right Model Format**
|
||
|
||
Selecting the correct model format depends on your **hardware capabilities** and **memory constraints**.
|
||
|
||
### **BF16 (Brain Float 16) – Use if BF16 acceleration is available**
|
||
- A 16-bit floating-point format designed for **faster computation** while retaining good precision.
|
||
- Provides **similar dynamic range** as FP32 but with **lower memory usage**.
|
||
- Recommended if your hardware supports **BF16 acceleration** (check your device’s specs).
|
||
- Ideal for **high-performance inference** with **reduced memory footprint** compared to FP32.
|
||
|
||
📌 **Use BF16 if:**
|
||
✔ Your hardware has native **BF16 support** (e.g., newer GPUs, TPUs).
|
||
✔ You want **higher precision** while saving memory.
|
||
✔ You plan to **requantize** the model into another format.
|
||
|
||
📌 **Avoid BF16 if:**
|
||
❌ Your hardware does **not** support BF16 (it may fall back to FP32 and run slower).
|
||
❌ You need compatibility with older devices that lack BF16 optimization.
|
||
|
||
---
|
||
|
||
### **F16 (Float 16) – More widely supported than BF16**
|
||
- A 16-bit floating-point **high precision** but with less of range of values than BF16.
|
||
- Works on most devices with **FP16 acceleration support** (including many GPUs and some CPUs).
|
||
- Slightly lower numerical precision than BF16 but generally sufficient for inference.
|
||
|
||
📌 **Use F16 if:**
|
||
✔ Your hardware supports **FP16** but **not BF16**.
|
||
✔ You need a **balance between speed, memory usage, and accuracy**.
|
||
✔ You are running on a **GPU** or another device optimized for FP16 computations.
|
||
|
||
📌 **Avoid F16 if:**
|
||
❌ Your device lacks **native FP16 support** (it may run slower than expected).
|
||
❌ You have memory limtations.
|
||
|
||
---
|
||
|
||
### **Quantized Models (Q4_K, Q6_K, Q8, etc.) – For CPU & Low-VRAM Inference**
|
||
Quantization reduces model size and memory usage while maintaining as much accuracy as possible.
|
||
- **Lower-bit models (Q4_K)** → **Best for minimal memory usage**, may have lower precision.
|
||
- **Higher-bit models (Q6_K, Q8_0)** → **Better accuracy**, requires more memory.
|
||
|
||
📌 **Use Quantized Models if:**
|
||
✔ You are running inference on a **CPU** and need an optimized model.
|
||
✔ Your device has **low VRAM** and cannot load full-precision models.
|
||
✔ You want to reduce **memory footprint** while keeping reasonable accuracy.
|
||
|
||
📌 **Avoid Quantized Models if:**
|
||
❌ You need **maximum accuracy** (full-precision models are better for this).
|
||
❌ Your hardware has enough VRAM for higher-precision formats (BF16/F16).
|
||
|
||
---
|
||
|
||
### **Summary Table: Model Format Selection**
|
||
|
||
| Model Format | Precision | Memory Usage | Device Requirements | Best Use Case |
|
||
|--------------|------------|---------------|----------------------|---------------|
|
||
| **BF16** | Highest | High | BF16-supported GPU/CPUs | High-speed inference with reduced memory |
|
||
| **F16** | High | High | FP16-supported devices | GPU inference when BF16 isn’t available |
|
||
| **Q4_K** | Low | Very Low | CPU or Low-VRAM devices | Best for memory-constrained environments |
|
||
| **Q6_K** | Medium Low | Low | CPU with more memory | Better accuracy while still being quantized |
|
||
| **Q8** | Medium | Moderate | CPU or GPU with enough VRAM | Best accuracy among quantized models |
|
||
|
||
|
||
## **Included Files & Details**
|
||
|
||
### `google_gemma-3-1b-it-bf16.gguf`
|
||
- Model weights preserved in **BF16**.
|
||
- Use this if you want to **requantize** the model into a different format.
|
||
- Best if your device supports **BF16 acceleration**.
|
||
|
||
### `google_gemma-3-1b-it-f16.gguf`
|
||
- Model weights stored in **F16**.
|
||
- Use if your device supports **FP16**, especially if BF16 is not available.
|
||
|
||
### `google_gemma-3-1b-it-bf16-q8.gguf`
|
||
- **Output & embeddings** remain in **BF16**.
|
||
- All other layers quantized to **Q8_0**.
|
||
- Use if your device supports **BF16** and you want a quantized version.
|
||
|
||
### `google_gemma-3-1b-it-f16-q8.gguf`
|
||
- **Output & embeddings** remain in **F16**.
|
||
- All other layers quantized to **Q8_0**.
|
||
|
||
### `google_gemma-3-1b-it-q4_k.gguf`
|
||
- **Output & embeddings** quantized to **Q8_0**.
|
||
- All other layers quantized to **Q4_K**.
|
||
- Good for **CPU inference** with limited memory.
|
||
|
||
### `google_gemma-3-1b-it-q4_k_s.gguf`
|
||
- Smallest **Q4_K** variant, using less memory at the cost of accuracy.
|
||
- Best for **very low-memory setups**.
|
||
|
||
### `google_gemma-3-1b-it-q6_k.gguf`
|
||
- **Output & embeddings** quantized to **Q8_0**.
|
||
- All other layers quantized to **Q6_K** .
|
||
|
||
|
||
### `google_gemma-3-1b-it-q8.gguf`
|
||
- Fully **Q8** quantized model for better accuracy.
|
||
- Requires **more memory** but offers higher precision.
|
||
|
||
|
||
# Gemma 3 model card
|
||
|
||
**Model Page**: [Gemma](https://ai.google.dev/gemma/docs/core)
|
||
|
||
**Resources and Technical Documentation**:
|
||
|
||
* [Gemma 3 Technical Report][g3-tech-report]
|
||
* [Responsible Generative AI Toolkit][rai-toolkit]
|
||
* [Gemma on Kaggle][kaggle-gemma]
|
||
* [Gemma on Vertex Model Garden][vertex-mg-gemma3]
|
||
|
||
**Terms of Use**: [Terms][terms]
|
||
|
||
**Authors**: Google DeepMind
|
||
|
||
## Model Information
|
||
|
||
Summary description and brief definition of inputs and outputs.
|
||
|
||
### Description
|
||
|
||
Gemma is a family of lightweight, state-of-the-art open models from Google,
|
||
built from the same research and technology used to create the Gemini models.
|
||
Gemma 3 1B model handles text only.
|
||
|
||
### Inputs and outputs
|
||
|
||
- **Input:**
|
||
- Text string, such as a question, a prompt, or a document to be summarized
|
||
- Total input context 32K tokens for the 1B size
|
||
|
||
- **Output:**
|
||
- Generated text in response to the input, such as an answer to a
|
||
question, analysis of image content, or a summary of a document
|
||
- Total output context of 8192 tokens
|
||
|
||
|
||
# <span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>
|
||
|
||
Please give like a click ❤️ . Also I’d really appreciate it if you could test my Network Monitor Assistant at 👉 [Network Monitor Assitant](https://readyforquantum.com).
|
||
💬 Click the **chat icon** (bottom right of the main and dashboard pages) . Choose a LLM; toggle between the LLM Types TurboLLM -> FreeLLM -> TestLLM.
|
||
|
||
### What I'm Testing
|
||
I'm experimenting with **function calling** against my network monitoring service. Using small open source models. I am into the question "How small can it go and still function".
|
||
🟡 **TestLLM** – Runs **Phi-4-mini-instruct** using phi-4-mini-q4_0.gguf , llama.cpp on 6 threads of a Cpu VM (Should take about 15s to load. Inference speed is quite slow and it only processes one user prompt at a time—still working on scaling!). If you're curious, I'd be happy to share how it works! .
|
||
|
||
### The other Available AI Assistants
|
||
🟢 **TurboLLM** – Uses **gpt-4o-mini** Fast! . Note: tokens are limited since OpenAI models are pricey, but you can [Login](https://readyforquantum.com) or [Download](https://readyforquantum.com/download/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) the Quantum Network Monitor agent to get more tokens, Alternatively use the TestLLM .
|
||
🔵 **HugLLM** – Runs **open-source Hugging Face models** Fast, Runs small models (≈8B) hence lower quality, Get 2x more tokens (subject to Hugging Face API availability) |