51 lines
2.8 KiB
Markdown
51 lines
2.8 KiB
Markdown
---
|
|
base_model: google/gemma-7b-it
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
license: other
|
|
model_type: gemma
|
|
library_name: transformers
|
|
inference: false
|
|
---
|
|
|
|
## Google Gemma 7B Instruct
|
|
- **Model creator:** [Google](https://huggingface.co/google)
|
|
- **Original model:** [gemma-7b-it](https://huggingface.co/google/gemma-7b-it)
|
|
- [**Terms of use**](https://www.kaggle.com/models/google/gemma/license/consent)
|
|
<!-- description start -->
|
|
## Description
|
|
This repo contains GGUF format model files for [Google's Gemma 7B Instruct](https://huggingface.co/google/gemma-7b-it)
|
|
|
|
## Original model
|
|
- **Developed by:** [Google](https://huggingface.co/google)
|
|
|
|
### Description
|
|
Gemma is a family of lightweight, state-of-the-art open models from Google,
|
|
built from the same research and technology used to create the Gemini models.
|
|
They are text-to-text, decoder-only large language models, available in English,
|
|
with open weights, pre-trained variants, and instruction-tuned variants. Gemma
|
|
models are well-suited for a variety of text generation tasks, including
|
|
question answering, summarization, and reasoning. Their relatively small size
|
|
makes it possible to deploy them in environments with limited resources such as
|
|
a laptop, desktop or your own cloud infrastructure, democratizing access to
|
|
state of the art AI models and helping foster innovation for everyone.
|
|
|
|
## Quantizon types
|
|
| quantization method | bits | size | description | recommended |
|
|
|---------------------|------|----------|-----------------------------------------------------|-------------|
|
|
| Q3_K_S | 3 | 3.68 GB | very small, high quality loss | ❌ |
|
|
| Q3_K_L | 3 | 4.4 GB | small, substantial quality loss | ❌ |
|
|
| Q4_0 | 4 | 4.81 GB | legacy; small, very high quality loss | ❌ |
|
|
| Q4_K_M | 4 | 5.13 GB | medium, balanced quality | ✅ |
|
|
| Q5_0 | 5 | 5.88 GB | legacy; medium, balanced quality | ❌ |
|
|
| Q5_K_S | 5 | 5.88 GB | large, low quality loss | ✅ |
|
|
| Q5_K_M | 5 | 6.04 GB | large, very low quality loss | ✅ |
|
|
| Q6_K | 6 | 7.01 GB | very large, extremely low quality loss | ❌ |
|
|
| Q8_0 | 8 | 9.08 GB | very large, extremely low quality loss | ❌ |
|
|
| FP16 | 16 | 17.1 GB | enormous, negligible quality loss | ❌ |
|
|
|
|
## Usage
|
|
You can use this model with the latest builds of **LM Studio** and **llama.cpp**.
|
|
If you're new to the world of _large language models_, I recommend starting with **LM Studio**.
|
|
<!-- description end --> |