57 lines
2.3 KiB
Markdown
57 lines
2.3 KiB
Markdown
|
---
|
||
|
|
tags:
|
||
|
|
- text-generation-inference
|
||
|
|
- transformers
|
||
|
|
- unsloth
|
||
|
|
- gguf
|
||
|
|
- reasoning
|
||
|
|
- Qwen2
|
||
|
|
- Qwen
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
[PRAM V2](https://huggingface.co/collections/Pinkstackorg/pram-v2-67612d3c542b9121bf15891c)
|
||
|
|
|
||
|
|
# 🧀 Which quant is right for you?
|
||
|
|
|
||
|
|
- ***Q4:*** This model should be used for super low end devices like older phones or older laptops due to its very compact size, quality is okay but fully usable.
|
||
|
|
- ***Q6:*** This model should be used on most modern devices, good quality and very quick responses.
|
||
|
|
- ***Q8:*** This model should be used on most modern devices Responses are very high quality, but its a little slower than q6
|
||
|
|
- ***BF16:*** This Lossless model should only be used if maximum quality is needed; it doesn't perform well speed wise, but text results are high quality.
|
||
|
|
|
||
|
|
## Things you should be aware of when using PARM models (Pinkstack Accuracy Reasoning Models) 🧀
|
||
|
|
|
||
|
|
This PARM is based on Qwen 2.5 0.5B which has gotten extra reasoning training parameters so it would have similar outputs to qwen QwQ (only much, smaller.), We trained with [this](https://huggingface.co/datasets/gghfez/QwQ-LongCoT-130K-cleaned) dataset.
|
||
|
|
it is designed to run on any device, from your phone to high-end PC. that is why we've included a BF16 quant.
|
||
|
|
|
||
|
|
|
||
|
|
To use this model, you must use a service which supports the GGUF file format.
|
||
|
|
Additionaly, this is the Prompt Template, it uses the qwen2 template.
|
||
|
|
```
|
||
|
|
{{{ if .System }}<|system|>
|
||
|
|
{{ .System }}<|im_end|>
|
||
|
|
{{ end }}{{ if .Prompt }}<|user|>
|
||
|
|
{{ .Prompt }}<|im_end|>
|
||
|
|
{{ end }}<|assistant|>
|
||
|
|
{{ .Response }}<|im_end|>
|
||
|
|
```
|
||
|
|
|
||
|
|
Or if you are using an anti prompt: <|end|><|assistant|>
|
||
|
|
|
||
|
|
Highly recommended to use with a system prompt.
|
||
|
|
|
||
|
|
# Extra information
|
||
|
|
- **Developed by:** Pinkstack
|
||
|
|
- **License:** apache-2.0
|
||
|
|
- **Finetuned from model :** unsloth/qwen2.5-0.5b-instruct-bnb-4bit
|
||
|
|
|
||
|
|
This model was trained using [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
|
||
|
|
|
||
|
|
Used this model? Don't forget to leave a like :)
|
||
|
|
|
||
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|