Update README with model details
This commit is contained in:
Glogwa
88
README.md
Normal file
88
README.md
Normal file
@@ -0,0 +1,88 @@
|
||||
---
|
||||
base_model: glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7
|
||||
library_name: gguf
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- fr
|
||||
tags:
|
||||
- granite
|
||||
- gguf
|
||||
- quantized
|
||||
- llama.cpp
|
||||
- ollama
|
||||
quantized_by: llama.cpp
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# granite-4.0-h-1b-DISTILL-glm-4.7-GGUF
|
||||
|
||||
GGUF quantized versions of [granite-4.0-h-1b-DISTILL-glm-4.7](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7)
|
||||
|
||||
## Available Formats
|
||||
|
||||
| Filename | Size | Quant Type | Description |
|
||||
|----------|------|------------|-------------|
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-f16.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-f16.gguf) | 2.73 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-F16 | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q2_k.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q2_k.gguf) | 0.55 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q2_K | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q3_k_l.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q3_k_l.gguf) | 0.71 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q3_K_L | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q3_k_m.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q3_k_m.gguf) | 0.68 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q3_K_M | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q3_k_s.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q3_k_s.gguf) | 0.65 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q3_K_S | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q4_0.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q4_0.gguf) | 0.81 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q4_0 | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q4_1.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q4_1.gguf) | 0.88 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q4_1 | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q4_k_m.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q4_k_m.gguf) | 0.84 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q4_K_M | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q4_k_s.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q4_k_s.gguf) | 0.81 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q4_K_S | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q5_0.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q5_0.gguf) | 0.96 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q5_0 | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q5_1.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q5_1.gguf) | 1.04 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q5_1 | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q5_k_m.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q5_k_m.gguf) | 0.98 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q5_K_M | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q5_k_s.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q5_k_s.gguf) | 0.96 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q5_K_S | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q6_k.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q6_k.gguf) | 1.12 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q6_K | |
|
||||
| [granite-4.0-h-1b-DISTILL-glm-4.7-q8_0.gguf](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF/blob/main/granite-4.0-h-1b-DISTILL-glm-4.7-q8_0.gguf) | 1.45 GB | GRANITE-4.0-H-1B-DISTILL-GLM-4.7-Q8_0 | |
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Ollama
|
||||
|
||||
```bash
|
||||
# Use Q4_K_M (recommended)
|
||||
ollama run hf.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF:Q4_K_M
|
||||
|
||||
# Or other quantizations
|
||||
ollama run hf.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF:Q8_0
|
||||
ollama run hf.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF:Q2_K
|
||||
```
|
||||
|
||||
### llama.cpp
|
||||
|
||||
```bash
|
||||
# Download and run
|
||||
llama-cli --hf-repo glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF --hf-file granite-4.0-h-1b-distill-glm-4.7-q4_k_m.gguf -p "Hello, how are you?"
|
||||
|
||||
# With server
|
||||
llama-server --hf-repo glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7-GGUF --hf-file granite-4.0-h-1b-distill-glm-4.7-q4_k_m.gguf -c 2048
|
||||
```
|
||||
|
||||
### LM Studio / GPT4All
|
||||
|
||||
Download the `.gguf` file of your choice and load it in your application.
|
||||
|
||||
## Quantization Details
|
||||
|
||||
| Type | Bits | Use Case |
|
||||
|------|------|----------|
|
||||
| Q2_K | 2 | Extreme compression, low quality |
|
||||
| Q3_K_M | 3 | Very compressed |
|
||||
| Q4_K_M | 4 | **Recommended** - Best size/quality |
|
||||
| Q5_K_M | 5 | High quality |
|
||||
| Q6_K | 6 | Very high quality |
|
||||
| Q8_0 | 8 | Near lossless |
|
||||
| F16 | 16 | Original precision |
|
||||
|
||||
## Original Model
|
||||
|
||||
This is the quantized version of [granite-4.0-h-1b-DISTILL-glm-4.7](https://huggingface.co/glogwa68/granite-4.0-h-1b-DISTILL-glm-4.7)
|
||||
|
||||
- **Base Model:** ibm-granite/granite-4.0-h-1b
|
||||
- **Fine-tuning Dataset:** TeichAI/glm-4.7-2000x
|
||||
- **Training Loss:** 0.6364
|
||||
Reference in New Issue
Block a user