75 lines
1.9 KiB
Markdown
75 lines
1.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: SupraLabs/Supra-50M-Instruct
|
|
library_name: llama.cpp
|
|
tags:
|
|
- gguf
|
|
- llama.cpp
|
|
- quantized
|
|
- supra
|
|
- text-generation
|
|
---
|
|
|
|
# Supra-50M-Instruct-GGUF
|
|
|
|
GGUF quantizations of [SupraLabs/Supra-50M-Instruct](https://huggingface.co/SupraLabs/Supra-50M-Instruct).
|
|
|
|
## Files
|
|
|
|
| File | Quantization | Size |
|
|
| --- | --- | ---: |
|
|
| `model.gguf` | F16 base GGUF | 100M |
|
|
| `Q8_0.gguf` | Q8_0 | 54M |
|
|
| `Q6_K.gguf` | Q6_K | 44M |
|
|
| `Q4_K_M.gguf` | Q4_K_M | 36M |
|
|
| `Q4_K_S.gguf` | Q4_K_S | 35M |
|
|
| `Q4_0.gguf` | Q4_0 | 33M |
|
|
| `Q3_K_S.gguf` | Q3_K_S | 30M |
|
|
| `Q2_K.gguf` | Q2_K | 28M |
|
|
|
|
Checksums are in `SHA256SUMS`.
|
|
|
|
## Conversion
|
|
|
|
`model.gguf` is the F16 GGUF from the source repository. The quantized files were generated with `llama.cpp` build `b9550` using `llama-quantize`.
|
|
|
|
Some K-quant outputs may contain fallback tensor types where tensor dimensions are not divisible by the required K-quant block size. This is normal `llama.cpp` behavior for this model shape.
|
|
|
|
## Usage
|
|
|
|
This model was instruction-tuned with the Alpaca prompt format. It is not a ChatML-style multi-turn chat model, so use completion mode and include the prompt template. The GGUF files intentionally do not include `tokenizer.chat_template`, because llama.cpp chat mode would otherwise try to apply the wrong template.
|
|
|
|
```text
|
|
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
|
|
|
### Instruction:
|
|
What is the capital of Japan?
|
|
|
|
### Response:
|
|
```
|
|
|
|
Example with `llama.cpp`:
|
|
|
|
```bash
|
|
cat > prompt.txt <<'EOF'
|
|
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
|
|
|
### Instruction:
|
|
What is the capital of Japan?
|
|
|
|
### Response:
|
|
EOF
|
|
|
|
llama-completion \
|
|
-hf di-zhang-fdu/Supra-50M-Instruct-GGUF:Q4_K_M \
|
|
-f prompt.txt \
|
|
-n 128 \
|
|
--temp 0.7 \
|
|
--top-k 50 \
|
|
--top-p 0.9 \
|
|
--repeat-penalty 1.15 \
|
|
-no-cnv
|
|
```
|
|
|
|
For deterministic checks, use `--temp 0 --top-k 1`.
|