初始化项目,由ModelHub XC社区提供模型
Model: eryk-mazus/polka-1.1b-chat-gguf Source: Original Platform
This commit is contained in:
48
README.md
Normal file
48
README.md
Normal file
@@ -0,0 +1,48 @@
|
||||
---
|
||||
base_model: eryk-mazus/polka-1.1b-chat
|
||||
inference: false
|
||||
language:
|
||||
- pl
|
||||
license: apache-2.0
|
||||
model_name: Polka-1.1B-Chat
|
||||
model_type: tinyllama
|
||||
model_creator: Eryk Mazuś
|
||||
prompt_template: '<|im_start|>system
|
||||
|
||||
{system_message}<|im_end|>
|
||||
|
||||
<|im_start|>user
|
||||
|
||||
{prompt}<|im_end|>
|
||||
|
||||
<|im_start|>assistant
|
||||
|
||||
'
|
||||
---
|
||||
|
||||
*I've copy-pased some information from TheBloke's model cards, hope it's ok*
|
||||
|
||||
For a model of this size, with stronger quantization, quality appears to decline much more than for larger models. Personally, I would advise to stick with `fp16` or `int8` for this model.
|
||||
|
||||
## Prompt template: ChatML
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
Jesteś pomocnym asystentem.<|im_end|>
|
||||
<|im_start|>user
|
||||
{prompt}<|im_end|>
|
||||
<|im_start|>assistant
|
||||
|
||||
```
|
||||
|
||||
## Example `llama.cpp` command
|
||||
|
||||
```shell
|
||||
./main -m ./polka-1.1b-chat-gguf/polka-1.1b-chat-Q8_0.gguf --color -c 2048 --temp 0.2 --repeat_penalty 1.1 -n -1 -p "<|im_start|>system\nJesteś pomocnym asystentem.<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant"
|
||||
```
|
||||
|
||||
Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
|
||||
|
||||
Change `-c 2048` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
|
||||
|
||||
If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
|
||||
Reference in New Issue
Block a user