98 lines
2.4 KiB
Markdown
98 lines
2.4 KiB
Markdown
---
|
||
license: mit
|
||
license_link: https://huggingface.co/microsoft/wavecoder-ultra-6.7b/blob/main/LICENSE
|
||
language:
|
||
- en
|
||
library_name: transformers
|
||
datasets:
|
||
- humaneval
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- code
|
||
metrics:
|
||
- code_eval
|
||
quantized_by: bartowski
|
||
---
|
||
|
||
## 4-bit GEMM AWQ Quantizations of wavecoder-ultra-1.1-6.7b
|
||
|
||
Using <a href="https://github.com/casper-hansen/AutoAWQ/">AutoAWQ</a> release <a href="https://github.com/casper-hansen/AutoAWQ/releases/tag/v0.2.4">v0.2.4</a> for quantization.
|
||
|
||
Original model: https://huggingface.co/microsoft/wavecoder-ultra-6.7b
|
||
|
||
Note: They updated the model in place, no idea if there's actually any changes, but tagging this 1.1 just in case
|
||
|
||
## Prompt format
|
||
|
||
No chat template specified so default is used. This may be incorrect, check original model card for details.
|
||
|
||
```
|
||
<|begin▁of▁sentence|>[INST] <<SYS>>
|
||
{system_prompt}
|
||
<</SYS>>
|
||
|
||
{prompt} [/INST] <|end▁of▁sentence|>
|
||
```
|
||
|
||
## AWQ Parameters
|
||
|
||
- q_group_size: 128
|
||
- w_bit: 4
|
||
- zero_point: True
|
||
- version: GEMM
|
||
|
||
## How to run
|
||
|
||
From the AutoAWQ repo [here](https://github.com/casper-hansen/AutoAWQ/blob/main/examples/generate.py)
|
||
|
||
First install autoawq pypi package:
|
||
|
||
```
|
||
pip install autoawq
|
||
```
|
||
|
||
Then run the following:
|
||
|
||
```
|
||
from awq import AutoAWQForCausalLM
|
||
from transformers import AutoTokenizer, TextStreamer
|
||
|
||
|
||
quant_path = "models/wavecoder-ultra-1.1-6.7b-AWQ"
|
||
|
||
# Load model
|
||
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
|
||
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
|
||
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
||
|
||
prompt = "You're standing on the surface of the Earth. "\
|
||
"You walk one mile south, one mile west and one mile north. "\
|
||
"You end up exactly where you started. Where are you?"
|
||
|
||
chat = [
|
||
{"role": "system", "content": "You are a concise assistant that helps answer questions."},
|
||
{"role": "user", "content": prompt},
|
||
]
|
||
|
||
# <|eot_id|> used for llama 3 models
|
||
terminators = [
|
||
tokenizer.eos_token_id,
|
||
tokenizer.convert_tokens_to_ids("<|eot_id|>")
|
||
]
|
||
|
||
tokens = tokenizer.apply_chat_template(
|
||
chat,
|
||
return_tensors="pt"
|
||
).cuda()
|
||
|
||
# Generate output
|
||
generation_output = model.generate(
|
||
tokens,
|
||
streamer=streamer,
|
||
max_new_tokens=64,
|
||
eos_token_id=terminators
|
||
)
|
||
```
|
||
|
||
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
|