commit 4c2a0710b7df865930c0c7d91e5c98f1ddc3ad23 Author: ModelHub XC Date: Sun Aug 9 06:09:16 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: twinkle-ai/gemma-3-4B-T1-it-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..931a6dc --- /dev/null +++ b/.gitattributes @@ -0,0 +1,41 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +gemma-3-4b-t1-it-q8_0.gguf filter=lfs diff=lfs merge=lfs -text +twinkle-ai-gemma-3-4B-T1-it-BF16.gguf filter=lfs diff=lfs merge=lfs -text +twinkle-ai-gemma-3-4B-T1-it-F16.gguf filter=lfs diff=lfs merge=lfs -text +twinkle-ai-gemma-3-4B-T1-it-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text +gemma-3-4b-t1-it-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text +twinkle-ai-gemma-3-4b-t1-it-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..1c15b30 --- /dev/null +++ b/README.md @@ -0,0 +1,265 @@ +--- +license: gemma +language: +- en +- zh +base_model: twinkle-ai/gemma-3-4B-T1-it +library_name: transformers +tags: +- Taiwan +- SLM +- GGUF +- agent +datasets: +- lianghsun/tw-reasoning-instruct +- lianghsun/tw-contract-review-chat +- minyichen/tw-instruct-R1-200k +- minyichen/tw_mm_R1 +- minyichen/LongPaper_multitask_zh_tw_R1 +- nvidia/Nemotron-Instruction-Following-Chat-v1 +metrics: +- accuracy +model-index: +- name: gemma-3-4B-T1-it + results: + - task: + type: question-answering + name: Single Choice Question + dataset: + name: tmmlu+ + type: ikala/tmmluplus + config: all + split: test + revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c + metrics: + - type: accuracy + value: 47.44 + name: single choice + - task: + type: question-answering + name: Single Choice Question + dataset: + name: mmlu + type: cais/mmlu + config: all + split: test + revision: c30699e + metrics: + - type: accuracy + value: 59.13 + name: single choice + - task: + type: question-answering + name: Single Choice Question + dataset: + name: tw-legal-benchmark-v1 + type: lianghsun/tw-legal-benchmark-v1 + config: all + split: test + revision: 66c3a5f + metrics: + - type: accuracy + value: 44.18 + name: single choice +pipeline_tag: text-generation +--- + +# Gemma 3 4B T1-it GGUF Collection + +
+ + Discord + + + Hugging Face + + + + License + +
+ +GGUF quantized models converted from [twinkle-ai/gemma-3-4B-T1-it](https://huggingface.co/twinkle-ai/gemma-3-4B-T1-it) for use with llama.cpp. + +![Gemma3-4B-T1-it](https://cdn-uploads.huggingface.co/production/uploads/62b085e6a14cbd643867d561/wqX8cTImAyfmdLxrrGW4L.png) + +## About + +Gemma 3 4B T1-it is a small language model fine-tuned on Taiwan-focused datasets, supporting both English and Traditional Chinese. This repository provides multiple quantization formats optimized for different use cases. + +## Available Models + +| Model | Size | Use Case | +| ----- | ---- | -------- | +| `twinkle-ai-gemma-3-4B-T1-it-BF16.gguf` | Largest | Best quality, highest precision | +| `twinkle-ai-gemma-3-4B-T1-it-F16.gguf` | Large | High quality, good precision | +| `twinkle-ai-gemma-3-4B-T1-it-Q8_0.gguf` | Medium | Balanced quality and speed | +| `twinkle-ai-gemma-3-4b-t1-it-q4_k_m.gguf` | Smallest | Fastest inference, lower memory | + +## Quick Start + +### Option 1: Using Hugging Face Hub (Recommended) + +Install llama.cpp via Homebrew: + +```bash +brew install llama.cpp +``` + +Run inference directly from Hugging Face: + +```bash +llama-cli --hf-repo thliang01/gemma-3-4B-T1-it-Q8_0-GGUF \ + --hf-file gemma-3-4b-t1-it-q8_0.gguf \ + -p "Your prompt here" +``` + +Start as a server: + +```bash +llama-server --hf-repo thliang01/gemma-3-4B-T1-it-Q8_0-GGUF \ + --hf-file gemma-3-4b-t1-it-q8_0.gguf \ + -c 2048 +``` + +### Option 2: Build from Source + +#### Step 1: Clone llama.cpp repository + +```bash +git clone https://github.com/ggerganov/llama.cpp +cd llama.cpp +``` + +#### Step 2: Build llama.cpp + +Basic build (CPU only): + +```bash +LLAMA_CURL=1 make +``` + +**Hardware-specific build options:** + +- **NVIDIA GPU (Linux)**: + ```bash + LLAMA_CUDA=1 LLAMA_CURL=1 make + ``` + +- **Apple Silicon (Mac)**: + ```bash + LLAMA_METAL=1 LLAMA_CURL=1 make + ``` + +- **AMD GPU (ROCm)**: + ```bash + LLAMA_HIPBLAS=1 LLAMA_CURL=1 make + ``` + +#### Step 3: Run inference + +```bash +./llama-cli --hf-repo thliang01/gemma-3-4B-T1-it-Q8_0-GGUF \ + --hf-file gemma-3-4b-t1-it-q8_0.gguf \ + -p "Your prompt here" +``` + +#### Step 4: Start server (optional) + +```bash +./llama-server --hf-repo thliang01/gemma-3-4B-T1-it-Q8_0-GGUF \ + --hf-file gemma-3-4b-t1-it-q8_0.gguf \ + -c 2048 +``` + +## Advanced Usage + +### Choosing the Right Model + +Select a model based on your needs: + +- **Best Quality**: Use `BF16` or `F16` versions (requires more memory) +- **Balanced**: Use `Q8_0` version (recommended for most users) +- **Resource Constrained**: Use `q4_k_m` version (suitable for devices with limited memory) + +### Common Parameters + +- `-p "prompt"`: Your input text for the model to respond to +- `-c 2048`: Context length (maximum number of tokens that can be processed) +- `--hf-repo`: Hugging Face repository name +- `--hf-file`: Model file name to use + +### Adjusting Generation Parameters + +```bash +llama-cli --hf-repo thliang01/gemma-3-4B-T1-it-Q8_0-GGUF \ + --hf-file gemma-3-4b-t1-it-q8_0.gguf \ + -p "Your prompt here" \ + --temp 0.7 \ + --top-p 0.9 \ + --repeat-penalty 1.1 +``` + +Parameter explanations: + +- `--temp`: Temperature (0.0-2.0), higher values produce more random output +- `--top-p`: Nucleus sampling parameter (0.0-1.0) +- `--repeat-penalty`: Repetition penalty to avoid repetitive content + +## Model Information + +- **Base Model**: twinkle-ai/gemma-3-4B-T1-it +- **Languages**: English, Traditional Chinese +- **License**: Gemma +- **Format**: GGUF (converted via [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo)) + +### Training Data + +- Taiwan reasoning and instruction datasets +- Contract review and legal documents +- Multimodal and long-form content +- Instruction-following examples + +### Benchmarks + +- **TMMLU+**: 47.44% accuracy +- **MMLU**: 59.13% accuracy +- **TW Legal Benchmark**: 44.18% accuracy + +## Troubleshooting + +### Common Issues + +**Q: Getting out of memory errors?** + +A: Try using a smaller quantized version like `q4_k_m`, or reduce the context length parameter `-c`. + +**Q: How can I speed up inference?** + +A: + +1. Use GPU acceleration (add hardware-specific flags during compilation) +2. Choose a smaller quantized model (like `q4_k_m`) +3. Reduce context length + +**Q: What prompt format does the model support?** + +A: This is an instruction-tuned model. Use a clear instruction format, for example: + +```text +Please analyze the main clauses of the following contract: [contract content] +``` + +## Links + +- [Original Model](https://huggingface.co/twinkle-ai/gemma-3-4B-T1-it) +- [llama.cpp Documentation](https://github.com/ggerganov/llama.cpp) +- [GGUF Format Documentation](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) + +## Contributing + +If you have any questions or suggestions, please feel free to open a discussion in the Hugging Face repository. + +--- + +**Note**: On first run, llama.cpp will automatically download the model file from Hugging Face. Please ensure you have a stable internet connection. \ No newline at end of file diff --git a/gemma-3-4b-t1-it-q8_0.gguf b/gemma-3-4b-t1-it-q8_0.gguf new file mode 100644 index 0000000..ce239d5 --- /dev/null +++ b/gemma-3-4b-t1-it-q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:854a386230c7fdd4c15133188e959d4a0be53e64f84c1baf22b41cc7fdff44b4 +size 4843612288 diff --git a/twinkle-ai-gemma-3-4B-T1-it-BF16.gguf b/twinkle-ai-gemma-3-4B-T1-it-BF16.gguf new file mode 100644 index 0000000..9f2e291 --- /dev/null +++ b/twinkle-ai-gemma-3-4B-T1-it-BF16.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:603a24fd155cf7305c01f684ada5c55135fc65dbd65ded2a4951a24abff1fac7 +size 9110313152 diff --git a/twinkle-ai-gemma-3-4B-T1-it-F16.gguf b/twinkle-ai-gemma-3-4B-T1-it-F16.gguf new file mode 100644 index 0000000..e010de7 --- /dev/null +++ b/twinkle-ai-gemma-3-4B-T1-it-F16.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1d1a1e347a0067d4a35993017c54cf6874ab7dcff9363bef6d64d4440793cd90 +size 9110313152 diff --git a/twinkle-ai-gemma-3-4B-T1-it-Q8_0.gguf b/twinkle-ai-gemma-3-4B-T1-it-Q8_0.gguf new file mode 100644 index 0000000..88f6d7e --- /dev/null +++ b/twinkle-ai-gemma-3-4B-T1-it-Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eb92173a84b5a97fed84bc35b7ea6fdeba2ae842904d6a1936447c14061be386 +size 4843612352 diff --git a/twinkle-ai-gemma-3-4b-t1-it-q4_k_m.gguf b/twinkle-ai-gemma-3-4b-t1-it-q4_k_m.gguf new file mode 100644 index 0000000..c56ec57 --- /dev/null +++ b/twinkle-ai-gemma-3-4b-t1-it-q4_k_m.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:56e173514961512231dfd86099af8f53bb6a4a12663595deb8b1eb4363f4e5d1 +size 2867477888