base_model, pipeline_tag, license
| base_model | pipeline_tag | license | |
|---|---|---|---|
|
text-generation | apache-2.0 |
| Quant | Size | Description |
|---|---|---|
| Q2_K | 331.2 MB | Not recommended for most people. Very low quality. |
| Q2_K_L | 476.1 MB | Not recommended for most people. Uses Q8_0 for output and embedding, and Q2_K for everything else. Very low quality. |
| Q2_K_XL | 754.3 MB | Not recommended for most people. Uses F16 for output and embedding, and Q2_K for everything else. Very low quality. |
| Q3_K_S | 371.86 MB | Not recommended for most people. Prefer any bigger Q3_K quantization. Low quality. |
| Q3_K_M | 394.8 MB | Not recommended for most people. Low quality. |
| Q3_K_L | 415.18 MB | Not recommended for most people. Low quality. |
| Q3_K_XL | 545.0 MB | Not recommended for most people. Uses Q8_0 for output and embedding, and Q3_K_L for everything else. Low quality. |
| Q3_K_XXL | 823.21 MB | Not recommended for most people. Uses F16 for output and embedding, and Q3_K_L for everything else. Low quality. |
| Q4_K_S | 448.98 MB | Recommended. Slightly low quality. |
| Q4_K_M | 461.79 MB | Recommended. Decent quality for most use cases. |
| Q4_K_L | 571.91 MB | Recommended. Uses Q8_0 for output and embedding, and Q4_K_M for everything else. Decent quality. |
| Q4_K_XL | 850.11 MB | Recommended. Uses F16 for output and embedding, and Q4_K_M for everything else. Decent quality. |
| Q5_K_S | 518.4 MB | Recommended. High quality. |
| Q5_K_M | 525.84 MB | Recommended. High quality. |
| Q5_K_L | 617.41 MB | Recommended. Uses Q8_0 for output and embedding, and Q5_K_M for everything else. High quality. |
| Q5_K_XL | 895.61 MB | Recommended. Uses F16 for output and embedding, and Q5_K_M for everything else. High quality. |
| Q6_K | 593.88 MB | Recommended. Very high quality. |
| Q6_K_L | 665.75 MB | Recommended. Uses Q8_0 for output and embedding, and Q6_K for everything else. Very high quality. |
| Q6_K_XL | 943.96 MB | Recommended. Uses F16 for output and embedding, and Q6_K for everything else. Very high quality. |
| Q8_0 | 767.47 MB | Recommended. Quality almost like F16. |
| Q8_K_XL | 1.02 GB | Recommended. Uses F16 for output and embedding, and Q8_0 for everything else. Quality almost like F16. |
| F16 | 1.41 GB | Not recommended. Overkill. Prefer Q8_0. |
| ORIGINAL (BF16) | 1.41 GB | Not recommended. Overkill. Prefer Q8_0. |
Quantized using TAO71-AI AutoQuantizer. You can check out the original model card here.