97 lines
2.4 KiB
Markdown
97 lines
2.4 KiB
Markdown
---
|
||||
|
|
license: mit
|
|||
|
|
license_link: https://huggingface.co/microsoft/wavecoder-ultra-6.7b/blob/main/LICENSE
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
datasets:
|
|||
|
|
- humaneval
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- code
|
|||
|
|
metrics:
|
|||
|
|
- code_eval
|
|||
|
|
quantized_by: bartowski
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4-bit GEMM AWQ Quantizations of wavecoder-ultra-1.1-6.7b
|
|||
|
|
|
|||
|
|
Using <a href="https://github.com/casper-hansen/AutoAWQ/">AutoAWQ</a> release <a href="https://github.com/casper-hansen/AutoAWQ/releases/tag/v0.2.4">v0.2.4</a> for quantization.
|
|||
|
|
|
|||
|
|
Original model: https://huggingface.co/microsoft/wavecoder-ultra-6.7b
|
|||
|
|
|
|||
|
|
Note: They updated the model in place, no idea if there's actually any changes, but tagging this 1.1 just in case
|
|||
|
|
|
|||
|
|
## Prompt format
|
|||
|
|
|
|||
|
|
No chat template specified so default is used. This may be incorrect, check original model card for details.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
<|begin▁of▁sentence|>[INST] <<SYS>>
|
|||
|
|
{system_prompt}
|
|||
|
|
<</SYS>>
|
|||
|
|
|
|||
|
|
{prompt} [/INST] <|end▁of▁sentence|>
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## AWQ Parameters
|
|||
|
|
|
|||
|
|
- q_group_size: 128
|
|||
|
|
- w_bit: 4
|
|||
|
|
- zero_point: True
|
|||
|
|
- version: GEMM
|
|||
|
|
|
|||
|
|
## How to run
|
|||
|
|
|
|||
|
|
From the AutoAWQ repo [here](https://github.com/casper-hansen/AutoAWQ/blob/main/examples/generate.py)
|
|||
|
|
|
|||
|
|
First install autoawq pypi package:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
pip install autoawq
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Then run the following:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
from awq import AutoAWQForCausalLM
|
|||
|
|
from transformers import AutoTokenizer, TextStreamer
|
|||
|
|
|
|||
|
|
|
|||
|
|
quant_path = "models/wavecoder-ultra-1.1-6.7b-AWQ"
|
|||
|
|
|
|||
|
|
# Load model
|
|||
|
|
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
|
|||
|
|
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
|||
|
|
|
|||
|
|
prompt = "You're standing on the surface of the Earth. "\
|
|||
|
|
"You walk one mile south, one mile west and one mile north. "\
|
|||
|
|
"You end up exactly where you started. Where are you?"
|
|||
|
|
|
|||
|
|
chat = [
|
|||
|
|
{"role": "system", "content": "You are a concise assistant that helps answer questions."},
|
|||
|
|
{"role": "user", "content": prompt},
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
# <|eot_id|> used for llama 3 models
|
|||
|
|
terminators = [
|
|||
|
|
tokenizer.eos_token_id,
|
|||
|
|
tokenizer.convert_tokens_to_ids("<|eot_id|>")
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
tokens = tokenizer.apply_chat_template(
|
|||
|
|
chat,
|
|||
|
|
return_tensors="pt"
|
|||
|
|
).cuda()
|
|||
|
|
|
|||
|
|
# Generate output
|
|||
|
|
generation_output = model.generate(
|
|||
|
|
tokens,
|
|||
|
|
streamer=streamer,
|
|||
|
|
max_new_tokens=64,
|
|||
|
|
eos_token_id=terminators
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
|