commit 40a79529a1d7ec7bb30ab0709bc41465afe4974e Author: ModelHub XC Date: Mon Sep 7 17:13:16 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: daniloreddy/gemma-3-270m-it_GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..fc10e62 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,46 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +gemma-3-1b-it_Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-1b-it_Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-1b-it_Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-1b-it_Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-1b-it_Q8_0.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-270m-it_Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-270m-it_Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-270m-it_Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-270m-it_Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-270m-it_Q8_0.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-270m-it_fp16.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..b7afad3 --- /dev/null +++ b/README.md @@ -0,0 +1,61 @@ +--- +license: apache-2.0 +base_model: google/gemma-3-270m +tags: +- llama.cpp +- gguf +- quantized +- text-generation +- lightweight +- lmstudio +- jan +- cobalt +- text-generation-webui +--- + +# Gemma-3-270m-IT - GGUF High-Quality Quantizations + +This repository provides **GGUF** quantized versions of the [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) model, optimized for local execution using `llama.cpp` and compatible ecosystems. + +## 📌 Version Notes +All quantizations were generated from the official **FP16** weights. +- **Target:** Efficient execution on consumer hardware, mobile/edge devices, and systems with limited memory. +- **Performance:** The output quality (reasoning, coherence, and accuracy) is strictly dependent on the base model's parameter scale (270M). + +## 📊 Quantization Table + +| File | Method | Bit | Description | +| :--- | :--- | :--- | :--- | +| **fp16.gguf** | FP16 | 16-bit | **Original Weights.** No quantization applied. Maximum fidelity. | +| **Q8_0.gguf** | Q8_0 | 8-bit | **Near-lossless.** Practically identical to the original model with lower memory footprint. | +| **Q5_K_M.gguf** | Q5_K_M | 5-bit | **High Precision.** Minimizes quantization error for critical tasks. | +| **Q4_K_M.gguf** | Q4_K_M | 4-bit | **Recommended.** Best balance between speed and performance. | +| **Q4_K_S.gguf** | Q4_K_S | 4-bit | **Fast/Small.** Optimized for maximum throughput and low RAM usage. | + +## 🛠️ Technical Details +- **Quantization Date:** 2026-03-07 +- **Tool used:** `llama-quantize` (llama.cpp) +- **Method:** K-Quantization (optimized for AVX2/AVX-512 and modern GPU architectures). + +## 🚀 How to Use +# Start a local OpenAI-compatible server with a web UI: + +### llama.cpp (CLI) using model from HuggingFace +```bash +./llama-cli -hf daniloreddy/gemma-3-270m-it_GGUF:Q4_K_M -p "User: Hello! Assistant:" -n 512 --temp 0.7 +``` + +### llama.cpp (CLI) using downloaded model +```bash +./llama-cli -m path/to/gemma-3-270m-it_Q4_K_M.gguf -p "User: Hello! Assistant:" -n 512 --temp 0.7 +``` + +### llama.cpp (SERVER) using model from HuggingFace +```bash +./llama-server -hf daniloreddy/gemma-3-270m-it_GGUF:Q4_K_M --port 8080 -c 4096 +``` + +### llama.cpp (SERVER) using downloaded model +```bash +./llama-server -m /path/to/gemma-3-270m-it_Q4_K_M.gguf --port 8080 -c 4096 +``` \ No newline at end of file diff --git a/gemma-3-1b-it_Q5_K_M.gguf b/gemma-3-1b-it_Q5_K_M.gguf new file mode 100644 index 0000000..6f5e664 --- /dev/null +++ b/gemma-3-1b-it_Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7d8e0733618291f32a6f72d22744b9adb5dbd980a42baaeabca202fb2a4a13cd +size 851345824 diff --git a/gemma-3-270m-it_Q4_K_M.gguf b/gemma-3-270m-it_Q4_K_M.gguf new file mode 100644 index 0000000..0a49f89 --- /dev/null +++ b/gemma-3-270m-it_Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7f63e9539d58e689e63a28eda9fe11e099bf87c414dcb4ed8c8d0cd808c508e2 +size 253115008 diff --git a/gemma-3-270m-it_Q4_K_S.gguf b/gemma-3-270m-it_Q4_K_S.gguf new file mode 100644 index 0000000..17875c9 --- /dev/null +++ b/gemma-3-270m-it_Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9ef78d089fa8260d943b33a56ca60ded11870cc7cb1f30f6bfb7cc1cc1c18c0b +size 249889408 diff --git a/gemma-3-270m-it_Q5_K_M.gguf b/gemma-3-270m-it_Q5_K_M.gguf new file mode 100644 index 0000000..3e20d46 --- /dev/null +++ b/gemma-3-270m-it_Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d6d35b3e0613673a2e5256ee16ca24b9fb3a5535a45b0912807b2535d18498eb +size 260027008 diff --git a/gemma-3-270m-it_Q5_K_S.gguf b/gemma-3-270m-it_Q5_K_S.gguf new file mode 100644 index 0000000..3463d8b --- /dev/null +++ b/gemma-3-270m-it_Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8cac082f5517964990f2cc50bee7876e24924792e8f18a3848863be57dcfae88 +size 257999488 diff --git a/gemma-3-270m-it_Q8_0.gguf b/gemma-3-270m-it_Q8_0.gguf new file mode 100644 index 0000000..df2f89d --- /dev/null +++ b/gemma-3-270m-it_Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:67b9d2ab180c5068595f2786309b7bd633974c80410020bbcc38d1a067050f46 +size 291545728 diff --git a/gemma-3-270m-it_fp16.gguf b/gemma-3-270m-it_fp16.gguf new file mode 100644 index 0000000..cf0782c --- /dev/null +++ b/gemma-3-270m-it_fp16.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:903799d7cd964d0803bee3ec6ed2c2247f92589ace9f4307e969c998e21681d2 +size 542835328