40 lines
2.1 KiB
Markdown
40 lines
2.1 KiB
Markdown
|
---
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
- zh
|
||
|
|
tags:
|
||
|
|
- llama
|
||
|
|
---
|
||
|
|
|
||
|
|
**This model repository contains files in GGUF format for the Yi 34B LLaMA, compatible with LLaMA modeling, based on the work from the [chargoddard/Yi-34B-Llama](https://huggingface.co/chargoddard/Yi-34B-Llama) repository.**
|
||
|
|
|
||
|
|
Based of the work of chargoddard's:
|
||
|
|
|
||
|
|
- Tensors have been renamed to match the standard LLaMA.
|
||
|
|
- Model can be loaded without trust_remote_code, but the tokenizer can not.
|
||
|
|
|
||
|
|
**Converted & Quantized Files**
|
||
|
|
|
||
|
|
### Yi-34B-Llamafied Model Options
|
||
|
|
|
||
|
|
The following tables list the available Yi-34B-Llamafied model files with their respective quantization methods and characteristics.
|
||
|
|
|
||
|
|
**Key:**
|
||
|
|
- **Size**: File size relative to the original.
|
||
|
|
- **Quality Loss**: The amount of quality loss due to quantization.
|
||
|
|
|
||
|
|
| Q-Method | File Name | Size | Quality Loss | Recommended |
|
||
|
|
|----------|---------------------|--------|--------------------------------------|----------------------|
|
||
|
|
| Q2 | Yi-34B-Llama_Q2_K | Smallest | Extreme *(not recommended)* | |
|
||
|
|
| Q3 | Yi-34B-Llama_Q3_K_S | Very Small | Very High | |
|
||
|
|
| Q3 | Yi-34B-Llama_Q3_K_M | Very Small | Very High | |
|
||
|
|
| Q3 | Yi-34B-Llama_Q3_K_L | Small | Substantial | |
|
||
|
|
| Q4 | Yi-34B-Llama_Q4_K_S | Small | Significant | |
|
||
|
|
| Q4 | Yi-34B-Llama_Q4_K_M | Medium | Balanced | **Recommended** |
|
||
|
|
| Q5 | Yi-34B-Llama_Q5_K_S | Large | Low | **Recommended** |
|
||
|
|
| Q5 | Yi-34B-Llama_Q5_K_M | Large | Very Low | **Recommended** |
|
||
|
|
| Q6 | Yi-34B-Llama_Q6_K | Very Large | Extremely Low | |
|
||
|
|
| Q8 | Yi-34B-Llama_Q8_0 | Very Large | Extremely Low *(not recommended)* | |
|
||
|
|
|
||
|
|
Please choose the model that best suits your needs based on the size and quality loss trade-offs.
|